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ABSTRACT 


Introducing simulation training to ground officers for the first time, within an 
existing proven eurrieulum, presents a number of ehallenges and questions. The proper 
amount of simulation time to evaluate and train skill sets and how to introduee simulation 
into an existing eurrieulum are mostly unknown. We have eompleted two studies at the 
Naval Postgraduate Sehool (NPS). The first study examined the length of time and the 
most appropriate method for introdueing simulation training to a user. The seeond study 
eompared the use of the Close Combat Marines (CCM) Taetieal Deeision-Making 
Simulation (TDS) with the traditional method of training deeision-making ealled the 
Taetieal Deeision-Making Game (TDG). The TDS and TDG were used in a between- 
subjeets experimental design to examine the viability of eaeh with regard to their ability 
to evaluate several important military traits. We found that both the TDG and the TDS 
methods were useful in evaluating a partieipant’s leadership eharaeteristies and deeision- 
making ability. However, only the TDS was eapable of evaluating situational-awareness. 
Our results also address a novel way in whieh these two approaehes eould be eombined 
to amplify eaeh other’s potential in training of ground offieers and military personnel in 
general. 
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I. INTRODUCTION 


A. PROBLEM STATEMENT 

Traditional training of military ground officers in tactics has evolved over the 
decades into a proven crawl, walk, run approach. Students are first taught the tactics, 
techniques, and procedures (TTPs) in a classroom that they will later execute in the field. 
The second phase is rehearsal in a controlled environment. This is still one step removed 
from actual implementation. Finally, after they have mastered the TTPs, they are 
encouraged to apply this knowledge in a training session that, most closely, approximates 
the real world in hopes that the jump from training to real will be as small as possible. 
Training results over time have shown that this methodical approach is an effective way 
to learn complex skills and team tasks. However, military educators have struggled with 
important questions about introducing a new category of computer-based instruction into 
an otherwise proven curriculum. 

As technology has increased over the last 20 years, military educators have begun 
to incorporate various forms of simulation and computer-based training into the crawl 
and walk phases of ground officer training. School administrators and teachers chose this 
approach because it made practical sense and provided a safe environment in which to 
train. Using this particular form of training system, students can make mistakes with few 
detrimental consequences. Some studies have demonstrated that simulation technology 
can help students learn faster, acquire more skills during sessions, and retain the 
knowledge for a longer period. Some types of simulation training expose people to 
training situations that would otherwise be extremely difficult to implement. Real world 
training, such as artillery call-for-fire or close air support training, can be very costly in 
terms of the number of units involved, time, and amount of resources required. To train 
one officer in a call-for-fire mission or a close air support task, with real assets, requires 
artillery pieces firing and/or aircraft flying. In a virtual environment, it is possible to 
train many personnel in these domains at one time without the cost of real supporting 
assets. 
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These are only some of the important benefits that simulation ean offer to the 
military when real world environments are found to be eost prohibitive. Simulation 
should not be eonsidered the ultimate replacement for real world field exercises because 
there are benefits that real world training can offer the student that simulation cannot. 
However, it can bridge the gap left open when real world training cannot be 
accomplished for various reasons. 

Many studies have explored simulation training relative to aviators, but few have 
examined it specifically for use among ground officers. The challenge is not introducing 
it in the first place, but to identify the most effective way to introduce and use it in 
training practice. Young officers, who are in the early stages of their training, might 
benefit from a targeted application of simulation technology. These early stages could be 
the location in the training cycle where simulation methods could prove to be the most 
useful. 

This study will focus on a comparative evaluation of a traditional method and a 
new type of simulation training. We trust that these insights will help answer questions, 
as well as, offer guidelines to a military school that is considering the integration of 
simulation into its training cycle. The answers to these questions will doubtlessly 
influence the welfare of military trainees who will be a part of the real world 
engagements soon after the completion of their training regimens. 

B. RESEARCH QUESTIONS 

Here is a list of questions that provided much of the motivation for our work. 
These questions were not only of interest to us personally, but, also, to the U.S. Marine 
Corps’ Basic School at Quantico, Virginia: 

• How much time is needed to familiarize a student with the system before 
training can actually begin? 

• Can a Tactical Decision-Making Simulation (Close Combat Marines 
(CCM)) be used to evaluate leadership? 

• Are there objective measures of leadership, decision-making, and 
situational awareness that can be evaluated with a TDG and a TDS? 
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• What are the training differenees between using a TDS and a TDG? Do 
they train the same skills? If so, what is the amount of training time 
required for eaeh? 

• Do the TDG and TDS enhanee the ability of the user to visualize the battle 
spaee? 

• How many sessions are neeessary to obtain a partieular level of 
eompeteney in taetical deeision-making? 

• Should TDG and TDS training be mutually exclusive or should they be 
used in a combined training approach? 

• Should simulation be used in a controlled instructor-led environment or 
should students be allowed to exclusively explore and learn on their own? 

This research, and the subsequent studies, was the result of our desire to explore 
the previous concepts and questions. We addressed a selected set of research questions 
and elaborated upon them in this thesis. The remaining research questions were 
considered, but not fully explored in the document. 

C. SCOPE 

The scope of this thesis will focus on the previous research questions and the 
evaluation of leadership, decision-making, and situational awareness by using two 
training methodologies. We will seek to provide answers to military training commands 
about the use of tactical decision-making simulations in their training curricula. We will 
compare and contrast the tactical decision-making game (TDG) and the tactical decision¬ 
making simulation (TDS) to see which of these approaches is best able to improve the 
efficiency and the effectiveness of training in the military environment. 

D, ORGANIZATION OF THESIS 

This thesis is organized in the following chapters: 

• Chapter I - Introduction. This chapter provides the motivation for this 
work, the research questions asked, and the framework for pursuing 
answers to these questions. 

• Chapter II - Background. This chapter is a literature review of scientific 
studies and experiments that have contributed to the understanding of 
leadership, decision-making, and situational awareness. The analysis and 
introduction of the tactical decision-making game and tactical decision¬ 
making simulation were, also, included. We, also, examined literature on 


3 



a Tactical Decision-Making Simulation (TDS), called Close Combat 
Marines, whieh the U.S. Marine Corps is using in its introduetory offieer 
and non-eommissioned offieer training. 

• Chapter III - Experiments. We eondueted two experiments. The analysis 
of the data eolleeted in eaeh was the basis for all eonelusions from this 
work. 

• Chapter IV - Reeommendations and Future Work. This ehapter diseusses 
lessons learned from both experiments and offers suggestions for future 
experiments and studies. It, also, provides reeommendations for future 
versions of taetieal deeision-making simulations like CCM. 
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II. BACKGROUND 


A. INTRODUCTION 

There are many skills and attributes that young offieers must aequire and cultivate 
to become effective military leaders. Leadership, decision-making (DM), and situational 
awareness (SA) are three essential characteristics of good military leaders. It is, 
therefore, understandable that all military schools endeavor to train, cultivate, and 
evaluate their students on these skills. At the Basic School in Quantico, Virginia, Marine 
Corps lieutenants are taught the basics of infantry combat. The Basic School’s mission is 
to, “Train and educate newly commissioned or appointed officers in the high standards of 
professional knowledge, esprit-de-corps, and leadership required to prepare them for duty 
as company grade officers in the operating forces, with particular emphasis on the duties, 
responsibilities, and warfighting skills required of a rifle platoon commander” (U.S. 
Marine Corps The Basic School Website, 2007). 

All military officers must lead. For them to lead effectively, they must be able to 
make timely decisions and clearly understand the situations unfolding around them. This 
rationale led us to select and conduct a study of leadership, decision-making, and 
situational awareness when Tactical Decision-Making Games (TDGs) and Tactical 
Decision-Making Simulations (TDSs) are used as training methods. 

B, LITERATURE REVIEW 

In this section, we provide a review of literature that served as a basis and a 
starting point for our work. The topics include leadership, tactical decision-making, 
situational awareness, tactical decision-making game training practice, and tactical 
decision-making simulations. 

1, Leadership 

As J.M. Burns stated in his book entitled Leadership, “Leadership is one of the 
most common and least understood phenomena in the world” (as cited in Popper, 1996, 
p. 15). This subject has been the center of learned discussion for millennia. Thinkers 
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have debated its origin, nature, and the subtle differenees between what some eall 
leadership and others eall management. Leadership is a quality that many elaim to know 
when they see, but generally have difficulty describing. Perhaps this is because the 
subject is vast and has so many facets that require analysis, review, and research. The 
nature of this work will be to focus on how leadership may be measured and documented. 
If needed, those measures could be used as additional guidance when deciding what 
specialization should be suggested to young officers. The authors will examine only a 
few of the most prominent theories and methodologies for measuring leadership, both in 
the real world, as well as in virtual environments (VE)s. 

a. History of Leadership Thought and Doctrine 

Early leadership theory can be traced back to ancient civilizations in 
Egypt, Babylon, Eastern Asia, and Greece. The 6**' century B.C. Chinese writer, Eao- 
Tzu, described the qualities of “the wise leader... as selfless, hardworking, honest, able to 
time the appropriateness of actions, fair in handling conflict, and able to empower 
others... Aristotle argued in Politics that leaders were to help others seek virtue; they 
would do so by themselves being virtuous” (Antonakis, Sternberg, 2004, p. 101). One 
could easily argue that good leadership has eternal characteristics. If the qualities can be 
defined, captured, and, measured, could they be sufficient to determine if someone is a 
leader and, if so, one that is good or bad? 

Eeadership has had many definitions. J. Kotter, writing the in Harvard 
Business Review, says simply that leadership is “getting people to act without coercion” 
(as cited in Popper, 1996, p. 15). Internationally known professor and author, Henry 
Mintzberg, believes that “the manager (like a military commander) does many things: 
coordination, logistics, management of information, budgets, and so forth. One of their 
roles is leadership: motivating people to perform tasks to the best of their ability” (as 
cited in Popper, 1996, p. 15). These two definitions hint at a subtle difference between 
managers and those who lead. There are two competing philosophies when seeking to 
articulate the definition of leadership. Throughout history, great leaders exhibited certain 
qualities. How they obtained these qualities is the crux of this long-standing controversy. 
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Are these traits innate or are they learned? One belief is that any person, who so desires, 
may learn the traits assoeiated with effeetive leadership and implement them (^"learned 
traif approaeh). If he is able to do so, then he will be an effeetive leader. The other 
view is that these traits are not learned, but innate. This is the elassie argument that 
leaders are born, not made. The “horn not made” philosophy asserts that the 
quantifieation of leadership eharaeteristies are for the identifieation of leaders, not for the 
training of sueh. To measure the effeetiveness of a leader, one must identify and 
eategorize those traits/qualities, form a seale on whieh to measure eaeh of them, evaluate 
them individually, and finally examine them eolleetively. However, an additional 
element must be present in both oases. A person, who has these qualities, whether 
learned or born, must aspire to leadership. He must, also, want it. If not, he will avoid 
opportunities to lead and never realize his potential. 

b. Modern Military Leadership Training 

The U.S. military establishment asoribes to a learned-trait approach. In 
the military, and over the last 35 years of the all volunteer foroe, those who have entered 
servioe, espeoially in the offioer oorps, are assumed to have this desire to lead beoause 
they are self-seleoted. Desire is assumed, as well as the notion that leadership oan be 
taught. The U.S. Marine Corps Offioer Candidate Sohool even has this notion oodified in 
its mission statement: “...to eduoate, train, evaluate, and soreen offioer oandidates to 
ensure they possess the moral, intelleotual, and physioal qualities for oommissioning, and 
the leadership potential to serve suooessfully as oompany grade offioers in the operating 
foroes” (USMC Offioer Candidate Sohool website). Terms, sueh as “eduoate,” “train,” 
and “leadership potential” all imply that there exists a oapability to learn the tenants of 
leadership. Also, that there is some ability toward leading other people that oan be 
aoquired. The Marines have oompiled a list of fourteen leadership traits oonsidered to be 
essential elements that leaders of all organizational levels are enoouraged to aoquire and 
master. They are: Justioe, Judgment, Dependability, Initiative, Deoisiveness, Taot, 
Integrity, Enthusiasm, Bearing, Unselfishness, Courage, Knowledge, Loyalty, and 
Enduranoe. 
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Retired Army Lieutenant General Walter F. Ulmer Jr. argued, in a 
January/February 2006 article from Armor magazine, that leadership is not a collection of 
“traits, personality, ambition, intellect, energy, and personal goals,” but “in practical 
terms, the process of leading is best measured not by attributes or characteristics, but by 
the specifics of what leaders do” (Ulmer, 2006, p. 39). He goes on to discuss a study of 
four division commanders who had just completed tours of duty in Operation Iraqi 
Freedom (OIF). This study examined “behaviors that are crucial for contemporary leader 
effectiveness” (Ulmer, 2006, p. 39) In addition to this study. Lieutenant General Ulmer 
looked at several other historical studies and found that a number of specific behaviors 
continued to surface in leaders who were widely thought of as good. He compiled a list 
from a variety of different levels of leadership. It was refined by an Army War College 
(AWC) group of professors and students into a list containing thirty items as seen in 
Figure 1. These have been further refined in Figure 2 into a subset called, the “Big 12.” 


Leader Behavior Preferences (LBP) Worksheet 

(Behaviors that create a command climate (hat supports operational excellence 
arid motivates competent people to continue military service.) 

Specific leader behavior (for a division commander) 





1. Adapts quickly to new situations and requirements. 



I 


2. Understands and employs current Army and Joint doctrine. 





3. Keeps cool under pressure. 





4. Knows how and when to involve others in decisionmaking. 





5. Clearly explains missions, standards, and priorities. 




— 

6. Sees the big picture; provides context and perspective. 




7. Sets high standards without a “zero defects" mentality. 





8. Encourages initiative and welcomes new ideas. 





9. Backs up subordinates; confronts the boss if necessary. 





10. Is trustworthy; keeps promises or explains why he can t. 





11. Employs units in accordance with their capabilities. 





12. Can handle "bad news." 





13. Gets out of the headquarters and visits the troops. 





14. Coaches and gives useful feedback to subordinates. 





15. Sets a high ethical tone; demands honest reporting. 





16- Will share the risks and hardships of his soldiers. 





17. Knows how to delegate and not "micromanage.” 





18. Is consistent and predictable in his behavior. 





19. Shows respect and consideration for others of any rank. 





20. Puts mission and people ahead of his own career. 





21. Is approachable; listens to questions and suggestions. 





22. Can make tough, sound decisions on time. 





23. Shares the limelight; gives due credit to others. 





24. Senses unproductive policies and makes prompt adjustments. 





25- Builds and supports teamwork within staff and among units. 





26. Holds people accountable for their actions and results. 





27. Is more interested in doing good than looking good. 





28. Is fair; doesn't play favorites with units or people. 





29. Is positive, encouraging, and realistically optimistic. 





30. Write in: 






Figure 1. 

Note: This list may ba reproduced with credit to AWC Study "Leadership Lessons at Division Command 
Level-2004." 


Figure 1. Leader Behavior Preferences (LBP) Worksheet (From; Ulmer, 2006) 
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“BIG 12” 

At the top of the list: (In order of question number) 

# 3. Keeps cool under pressure. 

# 5. Clearly explains missions, standards, and priorities. 

# 6. Sees the big picture; provides context and perspective. 

# 22. Can make tough, sound decisions on time. 

Also particularly significant: (In order of question number) 

# 1. Adapts quickly to new situations and requirements. 

# 7. Sets high standards without a "zero defects" mentality. 

# 12. Can handle "bad news.” 

# 14. Coaches and gives useful feedback to subordinates. 

# 15. Sets a high ethical tone; demands honest reporting. 

# 17. Knows how to delegate and not “micromanage.” 

# 25. Builds and supports teamwork within staff and among units. 

# 29. Is positive, encouraging, and realistically optimistic. 

Figure 2 


Figure 2. The “Big 12” (From; Ulmer, 2006) 


Micha Popper, Professor of Psychology of Haifa University, Haifa, Israel, 
compares leadership in the military and management in the business world. Some might 
argue that these are two names for the same thing. However, Popper demonstrates that 
there is a qualitative difference between the perceptions of a leader in and outside the 
military context. In a 1981 study examining leadership styles of military officers in Viet 
Nam, Gabriel and Savage assert that military leadership in Vietnam was overly 
influenced by a business-like sterile managerial style and not in keeping with traditional 
military leadership principles. This over reliance, or commercial management 
techniques, led to disastrous consequences in Viet Nam (as cited by Popper, 1996). 
Popper describes two very different relationships that exist between the leaders and the 
led; 


1. The Transactional Leader - The transactional leader is the leader 
who assesses his team’s psychological needs and determines the li nk 
between their amount of effort and reward for that effort. He is 
“transactional” in the sense that he is a broker of tangible and intangible 
incentives that are used to persuade his followers to accomplish some 
goal. This is most associated with a successful businessperson who 
orchestrates a connection between corporate goals and team performance 
(Popper, 1996). 

2. The Emotional Leader - The emotional leader is quite different, 
though he may still have a transactional element to his nature. He leads 
primarily by eliciting positive emotional responses from his followers. 
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The emotional leader is one who “may be deseribed by images, such as 
charismatic, visionary, and inspirational” (Popper, 1996, p. 16). The 
emotional leader is generally more common in military environments 
where tangible incentives are less common and intangibles dominate. 
This is the type of leader who is capable of arousing emotions so strong 
that “people are even willing to sacrifice their lives for the leader (Popper, 
1996, p. 16). 


c. Leadership Studies 

(1) Measuring Military Development (MD). In the opening 
chapter of his book. The West Point Way of Leadership, retired Army Colonel Larry R. 
Donnithome describes West Point’s philosophy of leadership: 

At every Fortune 500 institution in America, people are taught ethics. At 
West Point, people are taught character...A leader of character has all the 
qualities we normally associate with leaders’ ambition, confidence, 
courage, intelligence, eloquence, responsibility, creativity, compassion 
and one thing more which we unfortunately overlook too frequently 
among civilian leaders: A leader of character is absolutely trustworthy, 
even in times of great stress, and can be depended upon to put the needs of 
others, the organization, the community above personal considerations, 
not now and then, or when the spirit moves him, or when it will look good 
on his resume, but in every instance (Donnithome, 1994). 

Paul Bartone, Scott Snook, and Tmeman Tremble (2002), 
embarked on a 4-year longitudinal study of West Point cadets (N=1143) that examined 
the influence of cognitive and personality variables on military leadership performance. 
Due to attrition of some 25% of the cadet class over the course of four years, the final 
number of cadets in the study was N=855. A cross-validated hierarchical multiple 
regression procedure was used to determine the factors upon entry that successfully 
predicted military development grades in upper-classmen. The following potential traits 
were identified as potential predictors of leader performance: 

• Spatial Judgment 

• Logical Reasoning 

• Social Judgment 

• Problem Solving 

• College Entrance Equivalency Rating (CEER) 
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In addition to the previous five predictors of leadership 
performance, the researchers, also, examined personality based on the Neuroticism, 
Extraversion, Openness, Agreeableness, and Conscientiousness (NEOAC) personality 
inventory (Costa & McCrae, 1985). The process used to evaluate these predictors was to 
first examine the potential affects of the varying demographic variables (sex, race, age) 
on upperclassman leader performance scores. Then, move further into analyzing the 
cognitive and personality variables. If correlations were not established, then that 
particular variable was dropped from further consideration (Bartone, Snook, Tremble, 
2002 ). 

The researchers examined demographic, cognitive, and personality 
variables with the following results: There was no correlation found between the 
demographic variables of race and age with leader performance scores; thus, these were 
dropped. There was a correlation with sex; thus, this variable was retained for further 
consideration. Of the cognitive variables considered in the study, problem solving and 
spatial judgment (mental figure rotation) did not correlate with leader performance 
scores. These were, also, dropped from further consideration. Eogical reasoning, social 
judgment, and CEER did correlate significantly with leader performance and were kept 
for additional analysis. Of the personality variables, neuroticism and openness did not 
correlate with leader performance scores and were not analyzed further. Extraversion, 
agreeableness, and conscientiousness did correlate with leader performance; thus, they 
were kept (Bartone, Snook, Tremble, 2002). After the preliminary correlations, the 
following variables remained for consideration: sex, logical reasoning, social judgment, 
CEER score, extroversion, agreeableness, and conscientiousness. Based on a multiple 
regression analysis, the authors state that the previously mentioned demographic, 
cognitive, and personality variables must be considered predictive of leadership 
performance over time. After two hierarchical multiple regression analyses were done on 
these data, all variables were eliminated, leaving sex, college entrance scores, social 
judgment, and conscientiousness as predictors of leadership performance. 
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(2) Popper’s Big Three and Their Subeategories. Professor 
Mieha Popper asserts that there are three types of psyehologieal eapaeities that are 
essential for leadership; self-eonfidenee, a proaetive orientation, and a eapaeity for pro- 
soeial relationships. The three eapaeities eited here by Dr. Popper are based on the initial 
researeh of professor and author Bernard Bass (Bass, 1990). In the study published in 
Military Psyehology in 2004, Popper et al. administered questionnaires to 402 Israeli 
Defense Foree (IDF) soldiers from infantry and armor eorps units as they approaehed the 
end of their basie training. These were eompared with a soeio-metrie questionnaire 
examining the peer and eommander evaluations of the soldiers’ leadership eapaeities 
whieh, in-turn, were used to evaluate eaeh soldier and to elassify the soldiers either as 
leaders or non-leaders. Findings from this study revealed a signifieant differenee 
between leaders and non-leaders in all variables defined as psyehologieal eapaeities to 
lead (Popper, 2004). The researehers evaluated partieipants on the following levels: 

1. Loeus of Control (LOC) - refers to the eapaeity that one believes 
he has to eontrol events in his life. 

2. Self-Confidenee - eonsists of internal loeus of eontrol, low level of 
trait anxiety, and self-effieaey. 

3. Trait Anxiety - is eharaeterized by the degree to whieh a person 
will either exhibit, or not exhibit, symptoms of stress when faeed 
with a variety of situations. 

4. Self-effieaey - is defined as a person’s belief that he ean faee and 
aeeomplish a task or perform suoeessfully when presented with a 
goal. 

5. Proaetive Orientation - is measured by level of optimism. 

6. Capaeities Required for Pro-Soeial Relationships - is a measure of 
how well the person gets along with others. 

Popper’s findings resulted in the following eonelusions. From the 
above six measures, leaders were found to exhibit: 

1. Higher levels of internal loeus of eontrol when eompared to non¬ 
leaders. 

2. Lower levels of anxiety eompared with non-leaders 

3. Higher levels of self-effieaey eompared with non-leaders. 

4. Higher levels of optimism eompared with non-leaders. 
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5. Higher levels of secure attachment style compared with non 
leaders. 

6. Lower avoidant and anxious attachment styles compared with non¬ 
leaders (Popper, 2004, p. 257). 

d. Measuring Leadership in a Virtual Environment (VE) 

Leadership in a virtual environment is not that far removed from 
leadership in real environments. Many of the characteristics for evaluating leadership 
potential are essentially the same. However, research conducted by Steed, et al. in 1999, 
a group of researchers specializing in real time virtual environments from the University 
College in London, demonstrated that, in some cases, immersion within a virtual 
environment was positively correlated with leader behavior. 

In this paper, researchers discussed a series of three studies that 
investigated small group collaboration within a virtual environment. Groups consisted of 
three individuals who were tasked with solving a task that required a high degree of 
collaboration among the participants. The participants manipulated avatars that were 
colored red, green, and blue. They initially met in a virtual room and, then, were tasked 
with solving a series of puzzles that consisted of fragmentary statements that were written 
on pieces of paper hung on the walls in different parts of the virtual environment. The 
participants were required to locate all pieces of paper, and rearrange the phrases on each 
individual sheet to form an intelligible sentence. After the sessions, the researchers 
examined the relationships between the resources, presence, co-presence, immersion, 
group accord, and leadership. 

Study 1 - The first experiment was conducted with all participants located 
at the same university. In this study, two of the participants worked on desktop 
computers and the third participant was immersed via a head-mounted display (HMD). 
They started the task in a virtual environment and finished it in a real environment. 
There were ten groups of three that went through this study. The results suggested that 
the person who was immersed with the HMD tended to emerge as the leader for a 
significant amount of time. Additionally, group accord tended to be higher in the real 
meeting than in the virtual meeting. 
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Study 2 - The second experiment recreated the first study, but conducted 
the study over a wide area network. In this study, no real environment was present. All 
participants were placed in and remained in the virtual environment. No one participant 
was immersed more than the others and no participant emerged as a leader. Group 
accord matched the results of Study 1. 

Study 3 - In the final experiment, the two participants were physically 
located in London and Nottingham, England, and one in Greece. One participant was 
immersed more than the others via HMD. This was required to study whether use of a 
wide area network would have any effect on immersion. Dependent variables were self- 
reported presence as indicated via a questionnaire. Co-presence was, also, assessed via 
questionnaire. Group accord was measured similarly with a questionnaire. To account 
for a confound associated with personality, another question was added to control for 
anxiety effects resulting from the study. 

The results from a statistical analysis of the surveys found a positive 
correlation between presence and co-presence. Immersion had little or no effect at all on 
reported presence. Individual accord was positively correlated with co-presence. No 
clear pattern of leadership behavior emerged with respect to immersion. However, it was 
noted that the green participant (Greece) almost never emerged as the leader. This was 
likely a result of the wide area connection being 3-5 times slower than the network speed 
between London and Nottingham. Researchers, also, found that the immersed person 
with the HMD tended to emerge as the leader a significant amount of time. Steed, et ah, 
hypothesized that this was the case because he was presented with superior navigation 
metaphors (walking and moving naturally in space) and, therefore, he managed to reach 
more information about the environment than the others. That person, perhaps for the 
same reason, felt compelled to be more vocal about his findings and felt obligated to 
provide his teammates with the best information possible. Consequently, the same person 
was perceived as someone who knew what he was doing i.e., someone who was more 
likely to be perceived as a leader (talkativeness did correlate highly and positively with 
leadership). 
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e. Summary 

Many of the studies that we mentioned in this section found a number of 
factors believed to predict or, at least, to correlate with leadership. For instance, the U.S. 
Marine Corps publishes their fourteen leadership traits. Further, Lieutenant General 
Ulmer’s article in Armor magazine distills the leadership behaviors from four division 
commanders with combat experience in Iraq down to a list of thirty behaviors (Figure 1). 
Also, in Lieutenant Colonel Donnithorne’s book. The West Point Way of Leadership, he 
mentions nine leadership characteristics. Bartone, et ah, and Popper established a 
number of predictive leadership measures. 

A simple comparison yielded an interesting juxtaposition of 
traits/behaviors in Table 1 from Lao-tzu to modem military perspectives. The idea 
behind this table is to create a quick mechanism to compare each of the studies/lists 
mentioned in this work to identify if there is agreement between the different 
methodologies to identify leadership qualities (for either identification or predictive 
purposes). After the chart was established, we removed rows containing only one item. 
The purpose of this was to demonstrate agreement between the methods — not to list each 
line item represented. 
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Table 1. Comparative Analysis of Previously Mentioned Observations & Researeh 
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Confident and Calm Under Pressure 



♦ 

♦ 

♦ 


♦ 

Courageous 




♦ 

♦ 



Decisive 

♦ 


♦ 

♦ 




Eloquent and Articulate 



♦ 


♦ 



Encourages Initiative 

♦ 


♦ 

♦ 




Fair 

♦ 

♦ 

♦ 

♦ 




Intelligent 





♦ 

♦ 


Positive and Enthusiastic 



♦ 

♦ 



♦ 

Professionally Competent 



♦ 

♦ 




Selfiess and Loyal 

♦ 


♦ 

♦ 




Sound Judgment 



♦ 

♦ 


♦ 

♦ 

Trustworthy 

♦ 


♦ 

♦ 

♦ 




Most of these are a very elose mateh, but there was some subjeetivity in 
the plaeement and assignment of each leadership trait. Intelligence was not specifically 
mentioned by Ulmer, nor was it listed among the USMC leadership traits, but, from the 
other behaviors and traits that were listed, it becomes obvious that this was an important 
element of the leadership equation. Perhaps, this might be one that the authors believed 
was innate and individualistic; thus, it could not be acquired or improved upon as the 
other traits listed in the chart. These findings are interesting for a number of reasons and 
have obvious implications for military training. The NEOAC is such a simple instrument 
which could be used during initial military training to establish baselines of leadership 
potential. Virtual environments might be the next step in the evaluation of a student with 
respect to his leadership grade. This assumes that the VE was constructed to create 
situations where leadership could be demonstrated. We believe that additional focused 
studies should be conducted to confirm if leadership can be predicted based on specific 
observations of a user’s participation in a VE (Steed et ah, 1999). Conclusive evidence in 
this regard could provide the instructors with early indicators of how a student might 
perform in a real leadership situation. Though researchers are still unable to write a 
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definitive preseription for leadership, they are, at least, eloser to that goal than Lao-tzu 
and Aristotle were two to three millennia previously. Interestingly, the prineiples that 
these two philosophers saw so long ago in the leaders of their day are still both relevant 
and, in some eases, the same ones that philosophers today use to measure leadership. 

2. Decision-Making 

Deeision-making is an essential part of military operations. Modem seientifie 
research has developed numerous models of decision-making. Military organizations 
have incorporated some of these elements into their processes as well. The Military 
Decision-Making Process (MDMP) has been documented to assist military personnel 
with organizing their thoughts and preparing courses of action. Most recently, military 
schools have adopted computer-based decision-making simulations as a way to train 
small unit leaders in the art of tactical decision-making. 

a. The History of Military Decision-Making Doctrine 

Militaries throughout history have relied upon good decision-making 
methodologies to implement their tactics and strategies. The U.S. Army Field Manual 
(FM) 100-5 Operations defines decision-making as a combination of information coupled 
with use of judgment as an element of combat power. It emphasizes that decision¬ 
making requires knowing the following elements: If to decide, when to decide, and what 
to decide (as cited in Beal, 2002, p. 1). A method of decision-making is a critical skill 
from the highest echelons within the military establishment to the level of the individual 
soldier. To make quick course of action decisions, the leader must practice making 
decisions based on a consistent, logical, and deliberate process. 

Carl von Clausewitz, the famous Prussian military strategist of the early 
19* century, compares decision-making at the tactical and strategic levels: 

It takes much greater strength of will required to make key strategic 
decisions than a tactical one. With tactics, the actor is swept along by the 
moment and feels caught up in a whirlwind so intense that the struggle 
against it would result in the direst of consequences. With strategy, where 
everything moves much more slowly, there is plenty of room for one’s 
own misgivings, objections, and ideas - and those of others - and for 
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inopportune remorse. With strategy, one does not see at least half the 
situation with one’s own eyes; rather, everything must be guessed at and 
presumed, which decreases one’s level of conviction. As a result, most 
generals become bogged down with ineffectual fears when they should be 
taking action, (as cited in Ghyczy, von Oetinger, Bassford, 2001). 

Clausewitz clearly states that the tactical leader is required to make 
decisions faster than a leader at the strategic level. Decisions at the tactical level usually 
involve close to immediate life or death situations. For the leader at the tactical level to 
practice his decision-making, he must be in an environment where he must make 
decisions and experiences their consequences. Tactical decision-making simulations 
provide both the environment and the consequences for today’s military leaders. 

b. Decision-Making in Today’s Military 

Decision-making is not only a concept written about in the historical 
strategy books, but one that is crucial for victory. Military field manuals, related to 
strategy and tactics on the battlefield, require good decision-making processes. Chapter 2 
of FM 17-15 (1996), describes the concept of military command as having two vital 
components: decision-making and leadership. Furthermore, the same manual describes 
decision-making as a conscious process for selecting a course of action from two or more 
alternatives (FM 17-15, 1996). The importance of decision-making is, also, clearly seen 
in infantry platoon-level field manuals. FM 7-8, Infantry Rifle Platoon and Squad, states 
that decision-making in one of the most important skills that a leader can have. 
Furthermore, it states that the platoon and squad leaders should be tacticians and take the 
initiative on the battlefield. For them to make sound decisions, they must master 
knowledge of tactics, techniques, and procedures (TTPs) at the squad and platoon levels 
(FM 7-8, 2001, Chapter 1). 

The Military Decision-Making Process (MDMP) is an expedited logical 
set of guidelines that is used when there is more time to develop and plan for a course of 
action. MDMP is a method that can help commanders, and their staffs, to reach logical 
decisions in a timely manner. The commander, and the staff, determines different 
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courses of action for both friendly and enemy forces. After this phase, they develop their 
plan. The steps of the MDMP are contained in Army FM 101-5, Chapter 5, p. 3: 

1. Receipt of Mission 

2. Mission Analysis 

3. Course Of Action Development 

4. Course Of Action Analysis 

5. Course Of Action Comparison 

6. Course Of Action Approval 

7. Orders Production 

The MDMP is, also, addressed in detail in FM 5-0, Staff Organization, and 
Operations. It describes the deliberate form of MDMP as both a prescriptive process and 
an analytical tool (as cited in Van Poppel, 2005). The U.S. Marine Corps has a doctrinal 
publication (Marine Corps Warfighting Publication 5-1), named “Marine Corps Planning 
Process, that approximates the Army method contained in FM 101-5. 

c. Decision-Making Models 

There are many types of decision-making models within civilian and 
military applications. Military field manuals and civilian researchers perceive decision¬ 
making very similarly. They both understand it to be the process of choosing the most 
appropriate course of action among many. The two models that are most relevant to this 
work are the rational model and naturalistic decision-making. 

Rational Decision-Making: Oxford Business School Professor Loizos 
Heracleous (1994) explains rational decision-making as choosing the objective among a 
variety of alternatives. Heracleous also concludes that the applicability of this model is 
limited to relatively simple problems where objectives are clear, unambiguous, agreed 
upon, and cause-effect relations are well understood. This definition of decision-making, 
and the explanation of the rational model, is virtually identical to the definition stated in 
U.S. Army FM 101-5. Baxter et al. (2004) examined military decision-making and found 
that it was one of the eight dimensions of tactical thinking. Decision-making is the 
identification of a feasible course of action (COA) obtained from experience accumulated 
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in similar situations. Christopher Wickens et al. (2004) consider rational decision¬ 
making to be represented by three phases: 

1. Acquiring and perceiving information cues relevant for decision, 

2. Generating and selecting hypotheses, or situation assessments, 
about the meaning of the cues regarding the current state relevant 
to the decision, 

3. Planning and selecting the choices to take, on the basis of the 
inferred state, the costs, and the different outcomes. The three 
stages often cycle and iterate in a single decision. 

Naturalistic Decision-Making: Researchers soon discovered that humans 
often did not follow rational models for decision-making tasks. Instead, Gary Klein’s 
research on naturalistic decision-making in 1993, proposed that people use heuristic and 
individual bias from previous similar experiences to make decisions (as cited in Shattuck 
& Miller, 2006). Naturalistic decision-making seeks to describe how humans actually 
make decisions in the environment where the decisions would actually be made. 
Laboratory research of decision-making was considered artificial and ultimately an 
erroneous way to examine the real way that people make decisions. Furthermore, 
Christopher Wickens et al. (2004) state that people try to make the best possible decision 
for their given circumstances and within time limits. Instead of waiting for the very best 
solution to percolate to the top, they try to find the choice that is “good enough” for their 
purposes. This shortcut method is termed as “satisficing.” 

When we look at the military decision-making, we see that field manuals 
do not only propose the detailed and time consuming process to arrive at a good decision 
(MDMP), but, also, they promote a method of heuristics to evaluate the situation 
composed of the following elements: Mission, Enemy, Terrain, Troops, and Time 
(METT-T). METT-T permits a commander to obtain a relatively comprehensive mental 
picture of his particular situation without going through the extensive and time- 
consuming MDMP. The Marine Corps captures this particular concept in Marine Corps 
Doctrinal Publication 1-3 (Tactics) which states: 
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In some cases, speeding up the analytical decision-making process may be 
sufficient; however, in most cases, intuitive decision-making is needed to 
generate and maintain tempo. Intuitive decision-making relies on a 
commander’s intuitive ability to recognize the key element of a particular 
problem and arrive at the proper decision without having to compare 
multiple options (as cited in Nichols, p. 26). 

d. Decision-Making and Simulation 

Military organizations recognize that computer-based decision-making 
tools are important in the training of their personnel. One of the difficulties in acquiring 
good decision-making skills is that a leader needs to make many critical decisions. 
However, there may be few of these opportunities. For example, the opportunities to 
make tactical decisions in combat do not occur that often. The best decision-making 
environment that can be achieved in training is one that approximates the combat 
situation as closely as possible. Even training exercises are limited due to time, space, 
financial considerations, or physical limitations to conduct a large number of trials/runs 
in a given (usually short) period of time. One way of providing a decision maker with the 
opportunity to make many decisions is to introduce the use of tactical decision-making 
simulations (TDSs) in his regular training schedule. This can place a leader at a critical 
decision point many times. This method allows the leader not only to make decisions 
regarding how he would proceed with his own troops, but, also, to allow him to view the 
same situation from the enemy’s perspective. 

According to retired U.S. Marine Corps Gunnery Sergeant, Paul Nichols 
(2006), the focus is on enhancing Marines’ analytical decision-making abilities. There 
have been several studies in which scientists evaluated the training effectiveness of a 
single TDS. In their study, Baxter et al (2004) tried to evaluate how effective a TDS, 
called Close Combat Marines (CCM), was for the subjects compared to traditional paper- 
based tactical decision-making games (TDG). They did this study at the USMC Infantry 
Platoon Sergeant (IPS) Course, Advanced Training School, Camp Geiger, North 
Carolina. Fourteen USMC non-commissioned officers, who were attending the IPS 
course, participated in their study. The researchers administered surveys to the students 
and instructors. They required the students to choose a course of action (COA). Based 
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on the results of their data analysis, they concluded that the TDS (CCM) was most likely 
to address the macro-cognitive activities of uncertainty management and team 
coordination. The TDG better addressed mental simulation and planning. They could 
not find any significant evidence whether TDG or TDS was effective for training 
decision-making. 

There have been other studies that evaluated TDSs as well. In the Tactical 
Decision-Making Simulations II report, which was prepared by CHI Systems and Klein 
Associates for NAVAIR Orlando in 2004, the researchers tried to evaluate another TDS, 
named Marine Air Ground Task Force XXI (MAGTF XXI). The researchers stated that 
situational awareness is the foundation for making good decisions. In the beginning of 
the NAVAIR report, they hypothesized that SA accuracy scores would be higher in the 
TDG condition than in the TDS condition. This was confirmed by their findings. They, 
also, stated that follow-on research could investigate the best use of TDSs for producing 
improvements in SA and in assessing the impact of this training on decision-making. 

In their Naval Postgraduate School thesis. Games for Training: Leveraging 
Commercial Off The Shelf Multi-player Gaming Software for Infantry Squad Collective 
Training, Nolan and Jones (2005) sought to find out whether the first person shooter 
games could effectively train squads in collective, leadership tasks, and decision-making. 
They chose “combat drill” as the task that the experiment subjects would perform 
because it is a “collective action rapidly executed without applying a deliberate decision¬ 
making process” (as defined in FM 7-8, 2001, Chapter 4, p. 1). They concluded that the 
infantry squads can use commercial off the shelf (COTs) gaming software for this level 
and type of collective squad training. 

Military doctrine and manuals emphasize the importance of leadership and 
decision-making. Because they believe they are beneficial, military schools have adopted 
computer-based methods for training their young Commissioned Officers and Non- 
Commissioned Officers (NCOs). Modem scientific research confirms this notion and 
presents its own evidence that simulations have a strong role to play in this training 
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effort. TDSs offer the student the ability to aet as if he was leading a small unit in a real 
environment and present him with numerous opportunities to make deeisions in virtual 
eombat. 


3. Situational Awareness (SA) 

Situational awareness is a eoneept that has been the eenter of mueh aeademie 
diseussion. To some degree, this is a debate of semanties beeause most researehers 
aeknowledge that humans have some level of pereeption regarding the events that are 
taking plaee around them and that they use this information to make their deeisions. The 
eoneept of how SA is defined and measured in teams and in individuals is not only 
debated in eivilian eireles, but within military training environments as well. The 
remainder of this seetion will be devoted to the diseussion of eoneepts that are related to 
situational awareness. 


a. Definitions of Situational Awareness 

Although there have been many definitions of SA, Dr. Miea Endsley’s 
definition and her model are some of the most well known and widely aeeepted in the 
field. Endsley defines SA as “knowing what is going on around you” and “eonsisting of 
three levels: pereeiving elements in the environment within a volume of spaee and time; 
eomprehending what they mean in eontext; and predieting their status in the near future” 
(as eited in Stanners & Ereneh, 2005, p. 2). In a teehnieal paper written for the soeiety of 
automotive engineers, W.E. Hamilton (1987) provides an earlier definition of SA in the 
military environment as the knowledge of eurrent and near-term disposition of both 
friendly and enemy forees. The authors of a 1988 teehnieal report prepared for the 
department of defense foeused on four dimensions of SA ineluding spatial, identity, 
automation, and temporal awareness (Harwood et ah, 1988). 

b. The Need for SA or the Lack Thereof 

SA is essential for good deeision-making. A laek of it ean have disastrous 
eonsequenees. Dr. Endsley explains that “...problems with SA were found to be the 
leading eausal faetors in a review of military aviation mishaps, and in a study of aeeidents 
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among major air carriers, 88% of those involving human error eould be attributed to 
problems with situation awareness” (Endsley, 1999 , p. 1). Furthermore, though some of 
the aviation mishaps were found to be the result of ineorreet action selection, they are 
actually errors in SA (Endsley, 1999). The author stated that the aircrews made the 
correct decision based on their mental pieture of the situation, but, sinee that picture was 
inaccurate, they made erroneous deeisions that resulted in the mishap. 

The aviation eommunity is not the only group to have suffered from issues 
resulting from a laek of SA. There have been studies dealing with automobile driver SA 
as well. Ma & Kaber (2005) condueted a study regarding the effeets of an adaptive 
eruise eontrol (ACC) and use of a cell phone while driving. Half of their partieipants 
were required to use cell phones while driving virtual cars in a driving simulation with 
the use of ACC. The other half-used eell phones, but without the ACCs. The results 
indieated that use of the ACC system improved SA during the driving task and redueed 
driver mental workload. The cell phone eonversation caused negative effeets on driver 
SA and inereased driver mental workload. This study suggests that the introduetion of 
automation in the driving task frees up mental processes that the driver can bring to bear 
on other driving requirements. Therefore, automobile automation can significantly 
improve driver SA. The implieation of this study is that a laek of SA while operating a 
motor vehicle can increase the frequeney of aecidents. 

Not only is SA essential in aviation tasks and while operating automobiles, 
it is, also, an important part of ground eombat. Mathews et al. (2000) states that SA is a 
fundamental aspect of infantry operations and discusses a systematic approach to 
determine infantry SA requirements. Aecording to these researehers, the emerging 
infantry doetrine and non-linearization of the modem battlefield requires soldiers to 
quickly assess and act based on their available information. Finally, this study 
emphasizes that the ground military troops need good SA if they are to be sueeessful on 
today’s eomplex battlefield. 

c. Endsley’s Model And The Levels of SA 

Endsley and Garland (2000) defined SA in a three level model; 
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Level 1 - Perception: This is the foundational level on which the two 
subsequent levels rest. The perception of the status, attributes, and dynamics of the 
elements in the environment is the very first level of SA. Without basic knowledge of the 
environment, it is highly possible that the person will have an incorrect picture of his 
environment, which could lead to disastrous consequences. 

Level 2 - Comprehension: SA cannot be confined only to one’s 
perception of the world. It goes beyond simple awareness of the environment. This 
requires integration of different information from multiple sources. This level, also, 
requires mental processing of the information by choosing the right components based on 
the goal. 

Level 3 - Projection: This is the third and the highest level of SA. Level- 
3 SA is defined by Endsley as the ability to project forward from current events and to 
anticipate future occurrences. She emphasizes that this level allows the person to make 
rapid decisions by using his expectation of what might happen in the future. Figure 3 is 
Endsley’s Model of Situational Awareness in Dynamic Decision-Making in which all 
levels of SA are components. 
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Figure 3. Model of Situational Awareness (After: Endsley and Garland, 2000) 

In their book, Human Factors Engineering, Wickens et al. (2004), state 
that selective attention is an important element in perception and that projection heavily 
depends on working and long-term memory. 

d. SA and Decision-Making 

Endsley and Garland (2000) believe that SA is only a component of good 
decision-making as seen in Figure 3. They emphasize that SA is a precursor to decision¬ 
making and point out that there are other factors that influence SA and that can contribute 
to a good performance. Endsley (1995) explains that making an incorrect decision is 
possible even though there is perfect SA. Endsley gives an example of a military 
commander who has a perfect SA of the battlefield, but selects an inappropriate course of 
action (as cited in Endsley and Garland, 2000). 
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Dr. St. John et al. (2000), in a U.S. Navy SPAWAR technical report, 
explains the relationship between SA and tactical decision-making after they conducted 
two experiments. The subjects in their study were 28 Marine Corps officers and senior 
enlisted staff non-commissioned officers. 

In their first experiment, they hypothesized that “a good graphical 
representation leads to a better recall of enemy intent and future position than either 
textual or implicit representation of that information on a battlefield map” (St. John et ah, 
2000). The researchers provided this information to the participants under two methods. 
Method 1 was text descriptions of unit locations, size, and direction of movement. 
Method 2 showed participants representations of enemy location, size, and direction of 
movement on a high quality military grid map. After some time, the subjects were 
required to provide this information back to the researchers. The researchers did not find 
any statistical significance between the two methods of depicting the basic SA 
information. 

In their second experiment, they tried to answer the question; How does 
uncertainty affect tactical decision-making and how can its adverse effects be minimized 
or eliminated? Their hypothesis was that increasing situation uncertainty would lead to a 
more wait and see approach and, consequently, to a slower decision-making process. 
Furthermore, they thought that the effects of situation uncertainty would lead to longer 
decision-making times in inexperienced subjects versus experienced ones. Contrary to 
their expectation, rank and military experience did not predict the quality of their decided 
courses of action. The amount of experience that subjects had within a Combat 
Operations Center (COC) was the deciding factor with regard to the quality of their 
decided course of action. Their observation was that the participants who had less COC 
experience chose the wait-and-see approach, while the participants with more COC 
experience were found to be less affected by the uncertainty in the scenarios. 

Stanners and French (2005) did a study on the relationship between SA 

and decision-making. The number of participants was 24. They decided to use a 

commercial off-the-shelf first person shooter computer game called Operation 

Flashpoint. Participants were provided with a summary of a combat situation. The 
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researchers assessed SA by using the direct questioning technique (DQT). This consisted 
of asking questions to the participants at natural breaks during the game. SMEs were 
responsible for assessing decision-making among the participants. The SMEs scored the 
participants as optimal (5), average (3), or poor (1). This is a typical 1-5 Eikert scale. 
They used the action-inference decision tree (AIDT) to cover every possible action that a 
participant could take. The SME plotted the participant’s decision on the chart. They 
found a positive correlation between SA and decision-making, as well as, SA and 
planning. 


e. Measuring Individual SA 

The scientific community has different opinions regarding the exact 
method that should be used to measure SA and, as a result, they have developed several 
different ways to measure it. Salmon et al. (2004) classified different techniques to 
measure SA: 

• On-line freeze techniques. Direct measurement of the subject SA. 
Situational Awareness Global Assessment Technique (SAGAT) and 
Situational Awareness Rating Technique (SART) are the techniques that 
are widely validated. 

• Real-time probe techniques. There is no freeze of the task. This 
technique involves SA related queries during the task. 

• Self-Rating techniques. After the task, the subjects rate his/her own SA on 
a scale. 

• Observer rating techniques. There are subject matter experts (SMEs) who 
evaluate the SA. 

• Questionnaire techniques. SA-related questionnaires are provided for the 
subjects after the task. Then, the subjects evaluate themselves. 

SAGAT is the technique that was developed by Mica Endsley. Endsley et al. (2000) 
explains that the SAGAT procedure involves stopping exercises and simulations in a 
variety of ways; then, asking the participants questions that are designed to assess the 
individual’s SA. The questions incorporate perception, comprehension, and projection. 
Another popular measurement technique is SART. SART is generally used in the 
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aviation society. Endsley (1998) explains that SART provides subjective ratings of SA 
by the operator. SART ratings are highly eorrelated with operator workload and 
performance. 

The various SA assessment teehniques can all yield valid results. The 
proper tool/teehnique is a function of the type of task and the domain of the users 
involved. Based on the specifie situation, the researchers choose the teehnique. 

/. Measuring Team SA 

Many researeh teams and organizations have eonducted experiments on 
measuring individual SA in military environments. These evolved into a need to measure 
SA eolleetively within a team. Salmon et al. (2004) did a study foeused on finding the 
best measurement device for SA in a Command, Control, Computers, and 
Communieation (C4) environment. They indicated that the existing teehniques used to 
assess individual SA were adequate. However, there was a need for some meehanism to 
assess team SA. One of their conclusions was that a combination of techniques, sueh as 
the Situational Awareness Rating Seales (SARS) and SAGAT, would be a suitable way 
of assessing SA for C4 teams. Another study, entitled Development of a Metric for 
Collaborative Situation Awareness (CSA), focused on military SA. In this study. Redden 
et al. (2005) used PC-based game scenarios, whieh focused on actual military operations 
in urban terrain (MOUT) as their platform for the study. Their goals were to: 

• Design MOUT scenarios to elieit realistie communieations and tactics, 

• Develop a seript-based assessment of the team member’s and the team 
leader’s SA at the three levels of SA, 

• Manipulate eommunication struetures, 

• Assess the degree to which individual SA and the communication 
structures affeet the SA of the leader. 

The three-person teams used in the study played the roles of infantry 
squad leader and two subordinate team leaders. The researchers measured SA levels for 
both the team and squad levels. The questions that the researchers used to evaluate SA 
were designed to distinguish between the three levels of SA (Pereeption, Comprehension, 
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and Projection). The questions focused on the squad and fire-team levels of leadership. 
This study is a good example of how to look at team CSA using a simulation or a game- 
based environment. 

We can see that situational awareness is a vital component of decision¬ 
making within military aviation, as well as, for military ground training and combat. The 
military has several decision-making tools that might, also, be used for evaluating SA. 

4. Tactical Decision-Making Game (TDG) 

The tactical decision games (TDGs) are situational-based scenarios where 
individuals are required to exercise mental agility to meet the demands of 
the situational stimuli while implementing a problem solving solution. 

The TDG can range from paper media to a situation given orally to 
Marines by a seminar leader or facilitator. (Marine Corps Institute). 

Military organizations have been using TDGs for hundreds, even thousands of 
years. One of the earliest references to a TDG was from the famed Chinese general and 
theorist. Sun Tzu. “This technique (TDG) can be traced back at least to...Sun Tzu, who 
was advocating their use more than 2,500, years ago.” (U.S. Marine Corps, 1989). More 
recently, the Prussian state in the 1800s was noted to be using TDG. Many Prussian 
military leaders at the start of the 19* century believed that war was coming soon. Their 
military was faced with the monumental task of the state’s security and was required to 
quickly develop its officers and enlisted personnel. J.T. Gatto, in his 1991 book entitled 
The Prussian Connection, stated that the Prussian Army had to “prepare hard in 
peacetime to be ready when war began. From the very beginning of a Prussian (later 
German) cadet’s career, TDGs were used to sharpen the students’ decision-making skills 
and to provide a basis for evaluating them on their character” (as cited in Vandegrift, 
2006, p. 32). 

Modern TDGs are usually conducted with paper and pencil around a sand table or 
some other terrain model. They can, also, be conducted without a terrain representation. 
To improve situational awareness, the modern military uses TDGs primarily to build 
tactical decision-making skills and to assist military personnel with critical thinking 
(Gonsalves, 1997). 
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Some benefits of TDGs inelude: 

• Improving pattern reeognition skills, 

• Exereising the deeision-making proeess, 

• Improving and praetieing eommunieation skills, 

• Inereasing leadership potential. 

Some limitations of the TDG are: 

• Representing a snapshot in time, 

• Allowing participants to make only one move, 

• Simulating operating environment is difficult - impossible to simulate the 
friction and uncertainty of the operating environment, 

• Working for units no larger than company level, 

• Applying to special operations is difficult (Marine Corps Institute, 1989). 

5. Tactical Decision-Making Simulation (TDS) 

As computer technology became more prevalent and more powerful throughout 
the 1990s, TDGs began to give way to computer-based tactical decision-making 
simulations (TDSs). TDSs kept the basic premise of the TDG, but integrated computer 
technology with artificial intelligent agents. This new approach allowed the Marine, or 
team of Marines, to tight against the system that had an artificial intelligence component. 
Historically, the military has been one of the early adopters in the use of computer-based 
simulations applied in a training environment. However, there is still much to learn about 
how to best use them in a military training curriculum. 

Over the last two decades, all branches of the military service have begun to 
express more interest in using simulation to support the training of ground officers. 
Desktop simulations and digital game-based technologies have earned much attention for 
their potential as training interventions. Supporters view the sensory (predominantly 
visual and auditory) interactivity of the technologies as powerful means of fostering the 
development of cognitive skills relative to the task. 

Ground officers work in environments that are very fluid and dangerous; their 
tasks are less procedural and more subject to changing situations. TDSs allow them to 
experience near real-world tactical situations that offer the opportunity to make numerous 
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decisions without the risk of harm. Additionally, unlike a real-world training scenario, 
these decisions, and their consequences, can be replayed many times to the decision¬ 
maker as a training tool. This can illustrate what went right and what may have gone 
wrong. 

The research on TDSs has mostly dealt with overall teaching methodologies 
involving simulation, computer-based training, and commercial games used for training. 
The preponderance of the scientific literature on this subject has been devoted to macro- 
cognitive activities, decision-making, situational awareness, and training effectiveness. 
Researchers have focused on dealing mostly with overall learning and retention provided 
by the simulation (Baxter, Ross, Phillips, Shafer, and Fowlkes, 2004) and (Fowlkes, 
Cohn, Jones, Hafich, Nichols, Woodman, and Bushika, 2005). One area, in which many 
of these studies are lacking, is an emphasis on learning to use (operate) particular 
simulations. 

Some simulations require many hours to bring students to a level where they can 
operate them. This phase is the 'Hearn the system''’ phase, which is an essential piece of 
the overall training strategy. Since military training time is always limited, it is vital for 
students to learn how to use the simulation in the shortest period. This is so that they can 
begin progressing quickly toward the mastery of their learning objectives, instead of 
hunting and pecking their way through the session. This wastes time and resources. A 
lack of meticulous work in this area (examining preliminary training needs) could cause 
the introduction of TDS technology that does not capitalize on its training potential. 

In their Naval Postgraduate School thesis on leveraging commercial off the shelf 
games for use in military environments, Nolan and Jones (2005) conducted several 
training sessions. Initially they used the lecture method and, to a lesser degree, hands on 
computer control of their avatar (the virtual representation of a human within the game): 

We initially felt that the training we provided and their (the student’s) 
exposure to the game environment would be sufficient. Survey 
results...indicated that the participants felt that there was too much lecture 
and not enough hands-on training.. .Based on this feedback, we altered our 
training for the remaining two pilot studies. Our familiarization and 
training program evolved to a much shorter classroom presentation 
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followed by a group, hands-on training session in the virtual 
environment....The positive results from these pilot studies led us to use 
this ads our training and familiarization teehnique during the experiment 
eondueted at Ft Benning, GA (Nolan & Jones, 2005). 

Based on the previous researeh, we ean see that a variety of methods have been 
used to teaeh partieipants to use a simulation. However, the most effective one 
mentioned by these researchers is one where there is a short lecture phase preceding a 
“hands-on” training phase. This is where participants actually operate the simulation on 
their own. 


6, Comparing TDGs and TDSs 

The TDG and the TDS have many similarities, but there are a number of 
differences as well. Baxter et al. (2004) conducted a study that assessed the CCM TDS 
against the paper based tactical decision game (TDG). The researchers had fourteen 
participants from the Infantry Platoon Sergeant Course taught at the Advanced Infantry 
Training School at Camp Geiger, North Carolina. They used a within-subjects design, 
where all participants went through both the TDS and TDG. The researchers gave 
surveys to the students and instructors before and after the TDS and the TDG sessions. 
In the end, the researchers concluded that TDS was most helpful for developing the 
students’ mental coordination, mental simulation, and uncertainty management tasks; 
TDS was observed to be more helpful for planning. The researchers, also, examined 
decision-making and its relationship to the TDG/TDS. However, that proved 
inconclusive because neither TDS nor TDG was a recommended teaching method. More 
research is definitely needed to explore the ability of TDG and TDS to train and evaluate 
the skills that they are believed to train. 

7. Close Combat Marines (CCM) 

The Marine Corps has been leveraging aspects of digital-game based methods and 

inserting them into the tactical decision-making simulations (TDSs) to supplement 

existing training (Baxter, Ross, Phillips, Shafer, and Fowlkes, 2004, p. 1). The Marine 

Corps currently uses a TDS named Close Combat Marine (CCM) at various leadership 

and tactical training schools. This TDS is a top-down view of the battlefield and was 

33 



designed to develop the tactical decision-making skills of entry-level officers. The CCM 
used at the introductory infantry training school in Quantico, Virginia, is called The Basic 
School (TBS). TBS’s follow-on school for advanced infantry tactics is the Infantry 
Officer’s Basic Course (lOBC), also located in Quantico, Virginia, uses this product as 
well. CCM is not exclusively used for training U.S. Marine Corps officers. Enlisted 
Marines, also, gain exposure to this tool at the Platoon Leader’s Course (PEC) for non¬ 
commissioned officers (NCOs) at Camp Geiger, North Carolina (Nichols, 2006). 

CCM has a series of setup and information screens as seen in Eigure 4 



Eigure 4. CCM Startup, Mission, and Results screens 

The final screen displayed captures the results from the scenario and displays 
these to the TDS participants. Each CCM scenario comes with a recommended number 
of troop and vehicle assets. Each scenario is provided with two operations’ orders, one 
for the offense, and one for the defense. Einally, CCM provides a specific terrain that is 
unique to the scenario in which the teams engage one another and attempt to accomplish 
their missions. 
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III. EXPERIMENTS 


A. EXPERIMENT 1 - PRE-TRAINING STUDY LEARN THE SYSTEM 

1. Introduction 

Based on the findings from our literature review, we believe it was important to 
establish a base of knowledge among our participants. It is, also, important to provide 
them with an opportunity to learn how to operate the TDS system that will be used in our 
second study. To accomplish this, we designed a study to explore preliminary training 
approaches to see which method brought the users to an acceptable level of competency. 

In a U.S. Army Command and General Staff College Master’s Thesis entitled. 
The Application of Off-the-Shelf Military Simulations to Train Decision-Making and 
Teach Tactics, Shoemaker (2003) analyzed several different computer simulations. His 
study required officers to evaluate these simulations on their own time and report their 
findings back to him based on a template that was provided. While the specific results 
are interesting, they are not relevant to our work. What is interesting is that no 
preliminary training was provided at all. The officers were simply given the scenario and 
told to figure it out. Shoemaker reports that there was a variance of over four hours in the 
lengths of time that it took for each officer to evaluate the simulations (Shoemaker, 
2003). This report asserts that, given no initial training on how to best interact with the 
simulation, wide variations are possible in the lengths of time for a student to become 
familiar with a system or tool. It is likely that the time to learn to operate a system 
effectively is highly variable. It is, also, likely that this time period will considerably 
depend on the type of system that needs to be used in the study. 

In our initial study, we analyzed and identified a “best practices” approach to 
introducing to users a simulation called Close Combat Marines (CCM). Though CCM 
has been used as a training tool for approximately ten years in the U.S. Marine Corps, 
there is still a question about the best way of introducing it to new users. The intent of 
this experiment was to explore two issues: Firstly, the amount of time needed to learn to 
operate the simulation and, secondly, the best method for bringing a participant with no 
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experience with tactical decision-making simulations to a place where he would be 
competent and able to participate in initial TDS training. Additionally, we used this 
experiment to evaluate their experimental methodology to gain experience going through 
the institutional review board (IRB) process, experience system hardware and software 
setup, and practice the management of all participants throughout the entire process. 

2. Procedure 

We provided a very short (no more than five minutes) verbal explanation of the 
overall experimental scenario. The three methods in which the students participated 
were: 

• “Boot Camp Only” - The manufacturer provided a Boot Camp scenario 
designed specifically to train a user to operate the simulation; 

• “Memory Sheet Only” - A memory sheet that outlines all functionality of 
the simulation controls and describes the various parts of the screen in 
terms of menus and displays; 

• “Boot Camp Plus Memory Sheet” - A combined approach of the Boot 
Camp and the Memory Sheet. Our null hypothesis was that there was no 
difference between any of the three training methodologies. 

Participants engaged in and were evaluated on only one of the three above 
scenarios necessitating a between-subjects design. The following research questions 
were central to this investigation: 

• What is the best method to pre-train users to operate CCM effectively on 
their own? 

• Should this simulation always be used in a controlled instructor-led 
environment or should the students be allowed to explore and learn on 
their own? 

• How much simulation time is needed to accomplish this prior to the start 
of the actual training, i.e., how long should the introductory training 
sessions last? 

• How many training sessions are necessary before the user is ready for a 
real TDS training scenario? 

• Is a single training methodology adequate or should a combination of 
methodologies be used? 

Through this effort, we sought to obtain an improved set of introductory training 

approaches that will be incorporated into a future follow-on study. 
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Ho (Null Hypothesis) = There is no difference between the abilities of the three 
methods. 

Ha (Alternative Hypothesis) = There is some difference between the three 
methods with respect to the number of overall mistakes that each participant will make. 
We chose an « = 0.05 as the significance criteria whether to reject the null hypothesis. 

3, Method 

a. Participants 

The participants were all students of the Naval Postgraduate School 
(N=12), but ranged from among three different academic curricula. There were nine 
participants studying Human System Integration (HSI); one in Systems Engineering (SE); 
and two from the Modeling Virtual Environments and Simulation (MOVES) program. 
The HSI program is considered non-technical and the SE and MOVES degrees are both 
technical. All twelve participants were U.S. Naval officers (9 males and 3 females). 
Participants were randomly assigned to the three groups. A 3-block design was used to 
ensure even distribution to assigned groups. We, also, collected other demographic 
information, including age, dominant hand, and total time in service. Participants were 
all treated in accordance with ethical standards established by the American 
Psychological Association. Eurther, our study design was approved by the IRB 
committee at NPS. The IRB paperwork is included in Appendix B. 

b. Apparatus 

This experiment was conducted in the Human Systems Integration Eab 
(HISE) at the Naval Postgraduate School, Monterey, California. The layout of the space 
used during this study can be seen in Eigure 5. 
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(4) Lab Seats with Video 



Figure 5. Experiment 1 - Lab Setup 

We used four computer desktop workstations consisting of a CPU 
(Pentium 4), 19 inch monitor, a standard U.S. English keyboard, and a mouse. All of 
these workstations were wired into the schools network and had CCM installed on them. 
We used a Canon Digital Video Camcorder, with wide screen recording and lOx optical 
zoom, to record the screen that the participants were using throughout their test scenario. 
This camcorder captured audio of the participants for use in the verbal protocol analysis. 
Due to scheduling conflicts with other classes, we were only able to run one participant at 
a time through the experiment. Eor a single participant, we used two computers. On one 
computer, we had a standardized PowerPoint presentation that provided instructions 
specific to each of the three different training methodologies. On the other, they used 
CCM for the training and testing phases of the experiment. 

c. Procedure 

This study was a between-subjects 1x3 design. The independent variable 
was the method of training and the dependent variable was the number of mistakes 
participants made (based on analysis of their ‘think aloud’ protocol and the analysis of 
the audio/video recordings of their test sessions). We conducted the audio/video analysis 
jointly. 

The experiment lasted approximately 50-60 minutes for each participant. 

Upon arrival at the lab, participants were administered a survey that took demographic 
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information. Additionally, we asked the partieipants about prior experienee with CCM or 
other military tactieal deeision-making simulations. The survey required students to 
diselose their level of experience with commercial computer games and a number of 
other variables relative to their professional expertise. All participants received the same 
questionnaire, basic introduction to the experiment, and an overview of CCM. After this 
was completed, the participants were assigned to one of the three groups (we did not 
disclose the purpose of the experiment). 

(1) “Boot Camp Only.” The first group’s training scenario task 
was to learn to use the simulation only by exploring and going through the developer’s 
installed Boot Camp training. This segment of the CCM system was specifically 
designed by the simulation developers to provide new users with an easy way to learn 
how to move, shoot, issue orders, gather information regarding the health and welfare of 
their teams, as well as, to generally navigate within the virtual environment. No training 
aids were provided to the Boot Camp group. 

(2) “Memory Sheet Only.” Group 2’s training task was to 
learn to use the simulation by entering into and playing one of the scenarios provided 
with CCM. The users used CCM and were expected to learn its functionalities by using 
the system. Each person in this group was provided with a memory sheet that included 
all commands that they would need to operate CCM during the test phase (this sheet was 
created by summarizing the material from the CCM User Manual). The article, “Utility 
of Game Instruction” (Chen, 2003), inspired the use of a memory sheet. 

(3) “Boot Camp & Memory Sheet.” The training method used 
with Group 3 was to use a combination of the Boot Camp and the Memory Sheet. 

The time that subjects spent in each condition varied, to some degree, 
between participants because participants in training methods “Boot Camp Only” and 
“Boot Camp & Memory Sheet” were somewhat controlled by the sequential nature of the 
Boot Camp. The Memory Sheet Only method was much more free-form. Participants 
spent less time in this section of the experiment as they seemed eager to get to the test 
phase. All training sessions lasted no more than 20 minutes. If students expressed 
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confidence in their ability to operate the simulation prior to their 20-minute training 
session being completed, they were allowed to begin the test scenario at that time. All 
test scenarios lasted no more than 25 minutes. We stopped the scenario if the participant 
had not received a Mission Accomplished, or a Mission Failure notification by that time. 
Each participant went through the same test scenario “Module 3 Fight 3-la Machine Gun 
Defense.” Due to their actual performance during the scenario, the amount of time could 
be less than 40 minutes if the participant’s military forces called for or were offered a 
truce, were judged as absolutely victorious, or were annihilated by the artificial 
intelligence present with in the CCM system. 

We digitally recorded all sessions and required that participants verbalize 
their thoughts as they made their way through the instructions. They were asked to do 
the same during the actual use of the simulation during the test phase. This procedure 
was necessary to capture each participant’s intentions and to compare what the 
participants intended to do with what they actually accomplished. We, also, compared 
these results with information obtained in the Pre-TDS survey. Upon completion of their 
test session, participants were administered a Post-TDS survey in which they entered self- 
reported values to determine whether they felt they had learned specific CCM controls 
throughout the experiment. 

After the participants had departed, we reviewed over four hours of 
video/audio recordings and analyzed those capturing four different types of participant 
errors. We, also, conducted an overall assessment of their simulation strategies with 
respect to how they used the simulation. This was not an assessment of their tactical 
infantry decision-making prowess. Rather, it dealt strictly with the nature of the ability to 
control different aspects of the simulation and to get the information that they needed to 
make decisions. The most important factor that we analyzed was the participant’s ability 
to accomplish what he intended to after completing his particular method of training. 

4, Results 

Our primary focus was to determine which of those three training methods proved 
to yield the least errors during the test scenario. In Table 2, we summarize the results of a 
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descriptive analysis of the methods. The “Memory Sheet Only” group yielded the fewest 
total errors on average between the groups at 9.75. They, also, had the fewest errors 
when considering Error Factor at 2.6. 


Table 2. Summary Statistics 


1 

Boot Camp Only 

Memory Sheet 

Combination 
Boot Camp & 
Memory Sheet 

Total Errors (Mean) 

14.25 

9.75 

18 

Total Errors (Std Dev) 

10.59 

6.39 

4.83 

Error Factor (Mean) 

6.18 

2.6 

9.85 

Error Factor (Std Dev) 

6.05 

1.97 

6.97 


Error Factor = Total Errors/Instructor Evaluation 


Total Errors is a raw score that resulted from the simple tabulation of errors that 
the participants made during their testing phase. This, however, proved to be somewhat 
misleading because some participants were much more active in giving orders to their 
troops, telling them to move, shoot, ambush, and defend many more times than other 
participants. These active participants not only gave many more orders, but, in doing so, 
increased their chance of making errors. Consequently, we decided to assign a score that 
would take the activity level of the participants into account. We then divided the Total 
Number of Errors by this score, which resulted in the Error Factor. 

Inferential statistics yielded no significant difference between the three training 
methods. We used an F-Test (ANOVA) to test our hypothesis. This resulted in p = 
0.3563 for total errors and p = 0.2241 for Error Factor. Neither of these is sufficient to 
reject our null hypothesis (Ho = No difference between any of the methods). 

In addition to the primary hypothesis tests for the overall study, we did find some 
other interesting results. The participants who had a high level of previous commercial 
video game experience made significantly fewer errors. The sex of the participants, also, 
proved statistically significant: Males performed better than females. Finally, there was 
a significant result between gaming experience and sex indicating that males spent more 
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time playing video games than females. However, this study only had twleve participants 
(3 were female) and this low number could have significantly affected the results. 
(Detailed statistical output is provided in Appendix E at the end of this document). 

5. Summary 

The analysis of the data collected during the study led us to conclude that we were 
unable to reject our null hypothesis. However, there were other interesting findings that 
arose from a more detailed analysis of our data with respect to how other dependent 
variables influenced total errors and total errors with factor. One interesting finding, that 
caused us to come up with the factor in the first place, was that female participants were 
making very few errors. On the surface, it appeared that they were performing almost 
flawlessly. Upon scrutiny, however, the reason that so few errors were made was 
because very few commands were issued. They issued far fewer commands than males 
and, therefore, made fewer errors. 

There was one critical piece of data that we believe could have had significant 
influence on the total number of errors. However, the survey that solicited this 
information was ambiguous and confusing and was, therefore, removed from 
consideration. This variable was total amount of video game experience. It was not 
asked in terms of a single metric, such as number of hours/week. The question was 
structured in such a manner that the participants would likely not know whether they 
were answering in hours/day, hours/week, hours/month, or hours/year. This produced 
wild variations in the data from this question and, even with extensive effort to salvage 
the data; we decided that it would be too misleading to report it. 

It is interesting to note that the Memory Sheet method yielded the fewest errors in 
both the Total Error score, as well as, the Error Eactor. The Memory Sheet method was 
the method that allowed the participants to engage in a real scenario versus going through 
the simulation developer-provided boot camp. We noticed that when the memory sheet 
was provided in methods 2 and 3, it was mostly unused by participants. After the 
scenario, many of the participants mentioned they totally forgot the memory sheet as 
soon as they started taking fire from the enemy. We believe that the memory sheet was 
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mostly irrelevant to any learning that took plaee. Further, we believe that the Memory 
Sheet Only method was the best method for training beeause it most elosely aligned with 
how users would use the simulation during the test seenario. They were required to issue 
orders, view toolbars, and assess their teams in exaetly the same manner that they would 
be required to during the test phase. In effeet, they were praetieing how they would use 
the simulation. This important training eoneept was not present in either training method 
one or three where the boot eamp was the primary instruetional tool. 

Based on the information obtained in this study, we ean offer preliminary answers 
to our researeh questions: 

(1) What is the best method to pre-train users of the TDS to operate it 
effeetively on their own? The answer to this question was ineonelusive based on our 
findings. 

(2) Should this simulation always be used in a eontrolled instruetor-led 
environment or should the students be allowed to explore and learn on their own? We do 
not believe that the former should always be the ease. We found that instruetor-led 
training would be benefieial only as an introduetion to the TDS itself. This should be 
very brief After that, the student should be plaeed in front of a eomputer with the TDS 
installed and allowed to go through a simple module, sueh as Module 1 Fight 1-la, 
Maehine Gun Defense. The instruetor would be either beside or behind the student to 
provide immediate answers to the student’s questions on the eonduet of the TDS. This 
approaeh allows the student to explore the simulation seenario and forees him to use as 
many resourees as possible to aeeomplish his mission. 

(3) How mueh simulation time is needed to aeeomplish this prior to the start 
of the aetual training, i.e., how long should the introduetory training sessions last? The 
questionnaire results varied widely from “weeks” to “another 10-20 minutes.” The latter 
is probably eloser to the aetual answer. We believe that the absolute laek of prior gaming 
experienee from the partieipant, who stated “weeks” would be required to gain 
profieieney, was exaggerating due to obvious frustration with his laek of understanding 
and ability to aeeomplish his intentions throughout the test phase. 
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(4) Is a single training methodology adequate or should a eombination of 
methodologies be used? We believe that a eombination of methodologies should be used. 
Users should be introdueed with a brief overview of CCM via a leeture method 
aeeompanied by several carefully crafted PowerPoint slides depicting images of the IDS 
in various stages. The various screens could be introduced along with the overall goal of 
the simulation. This should last no more than 5-10 minutes. The learning process should 
consist of an instructor taking questions, but getting the users behind a station as quickly 
as possible. It is important to introduce them to a simple scenario and let them explore. 
This provides them with expert guidance and answers to their questions as they arise. 

(5) How many training sessions are necessary? On average, participants 
stated that only 1-2 sessions would be required to master the basics of operating the 
CCM controls. We agree with this estimate because the participants’ opinions were 
confirmed by our analysis of the data. 

One consideration for future work might be to examine some elements of 
personality, such as extroversion and openness to see if they are influential on learning to 
use the controls within a TDS. We were struck by how some participants were very 
timid with the interface and consistently stated how unsure they were of what to do and 
how to do it. Others took control immediately and began to issue orders to CCM teams 
in a chaotic and haphazard way, even telling friendly teams to fire on other friendlies. 
There was no comprehension about who was who or what the overall goal of the scenario 
was (even though it had been explained in great detail during the scenario setup phase). 

The results from the experiment were very influential when we designed 
experiment 2. We drew on lessons learned to develop a brief instructor-led introduction 
to the TDG/TDS and what could be expected in each session. We, then, allowed the 
students to engage in their particular roles as quickly as possible. We remained close by 
to answer questions. This was the intent of session 1 in both the TDG and the TDS. 
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B, EXPERIMENT 2 - PILOT STUDY EVALUATING LEADERSHIP, 

DECISION-MAKING, AND SITUATIONAL AWARENESS WITH TDG 

AND TDS 

This section describes our second experiment where our intent was to examine 
leadership, decision-making, and situational awareness in virtual environments and how 
this differed from the traditional TDG training method. 

1. Introduction 

Tactical decision-making games and simulations have been used for some time in 
the training of ground officers, but to what extent they could be used to evaluate 
leadership, decision-making, and situational awareness, is largely unknown. Some 
studies have explored decision-making and situational awareness with respect to TDSs, 
but even fewer examined the leadership question. The TDG has been used by 
professional military schools training junior commissioned officers and non¬ 
commissioned officers. This training method has been proven effective in evaluating 
decision-making based on the evaluations of subject matter experts. The Tactical 
Decision-Making Simulation (TDS), on the other hand, has only recently been introduced 
in some of these schools. There is very little knowledge about how to implement it, what 
it can accomplish, and whether or not it can replace or augment the current TDG training 
regimen. 

To evaluate the three characteristics mentioned above, we will use two different 
training methodologies; TDG and TDS. 

• Ho (Null Hypothesis) = There is no difference between the abilities of the 
TDG and TDS to train and evaluate military personnel with respect to 
decision-making, leadership, and situational awareness. 

• Ha (Alternative Hypothesis) = There is some difference between the TDG 
and TDS training methodologies with respect to their ability to train and 
evaluate decision-making, leadership, and situational awareness in 
military personnel. 
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We chose an a = 0.05 as the significance criteria whether to reject the null 
hypothesis. A 2-tailed t-test will be used to determine if the two training methods are 
different with respect to evaluating/training leadership, decision-making, and situational 
awareness. 

2. Procedure 

From Experiment 1, we learned lessons which we implemented in this study. The 
design of this experiment remained between-subjects. Additionally, we realized that a 
single training session would not be adequate to build the necessary skill level. Also, it 
would not be sufficient for participants to demonstrate characteristics of any of the 
dependent variables (Leadership, Decision-Making, and Situational Awareness). We 
decided on a single introductory session lasting approximately one hour to teach the TDS 
participants how to navigate and participate in a typical CCM scenario. The same 
amount of time was reserved for the TDG group. It was used to instruct them about a 
TDG session and how to participate in one. Infantry tactics, techniques, and procedures 
(TTP) instruction was provided to both groups during this first hour (same instruction for 
both groups). We were aware that unless participants had prior training in infantry TTPs, 
they would only be able to understand TTP instruction at the most basic level. However, 
this level was all that was required for them to participate effectively in either the TDG or 
the TDS groups. 

We made every effort to standardize the make-up of the teams in the TDG and the 
TDS. The TDG and the TDS groups consisted of a total of eight personnel each. Within 
both groups, there were two subgroups of four participants each. In both TDG 
subgroups, we assigned one of the four to act as the platoon leader; with the remaining 
three to act as squad leaders. Likewise, in the TDS subgroups, there was one platoon 
leader and three squad leaders. All participants functioned in their assigned roles 
throughout their sessions. 

The remaining four sessions were split evenly between offensive and defensive 
scenarios. The CCM software comes with a number of prepackaged training scenarios 
that allow for individual or team-on-team modes. The team-on-team modality was 
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selected for use in all of the remaining sessions (Sessions 2-5). The same built-in CCM 
scenarios were used by the TDG group to ensure that the IDS and TDG groups were 
presented with the same level of difficulty within their scenarios. The CCM scenarios 
changed from session to session which, consequently, altered the troop mix and terrain. 
Table 3 identifies which CCM scenario was used in each session and what terrain was 
presented to the participants. 


Table 3. TDG/TDS Sessions and Scenarios 


Day 

CCM Scenario 

Session 

TDG 

TDS 

Team 1 

Team 2 

Team 1 

Team 2 


1-la Town 

1 

Introduction 

Introduction 

Introduction 

Introduction 


5-lb Town 

2 

Defense 

Defense 

Offense 

Defense 


5-lb Town 

3 

Offense 

Offense 

Defense 

Offense 


2-lb Airfield 

4 

Defense 

Defense 

Offense 

Defense 


6-lb Farm 

5 

Offense 

Offense 

Defense 

Offense 


We collected two types of data: self-reported through questionnaires (pre and 
post-session) and evaluations by the instructors. Once each session had concluded, the 
participants were presented with questionnaires. The collected data was used later to 
determine relative leadership, decision-making and situational awareness scores. Most of 
these scores were perceived values based on the subjective assessments of each 
participant and his/her opinion of teammates’ performances within each subgroup. 
Additionally, the instructors evaluated the platoon leaders in each session from both the 
TDG and TDS groups. It is clear that situational awareness can only be evaluated for the 
TDS participants. However, based on TDG participants’ comments and discussions 
during sessions, we expected them to have a level of understanding about their 
teammates’ situational awareness. We extracted those SA answers from the 
questionnaires and used them to determine individual SA for the TDG group members. 

The procedure we used to determine SA for each participant was called the 
Geographical Recall and Analysis of Data in the Environment (GRADE) (Miller and 
Shattuck (2007)). An example of this tool can be found in Appendix M. In the middle of 
the TDS sessions, we paused the simulation and evaluated each participant’s knowledge 
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concerning the loeation, size, and direction of movement of the friendly and enemy 
forees. We did this by having them draw eireles and arrows on a map sheet. We walked 
between the rooms reeording the exact location, size, and direction of troop movement 
and used this as ground truth. All IDS partieipant GRADE sheets were evaluated based 
on this ground truth sheet. Use of this method was not possible for the TDG group 
beeause the TDG offers no ground truth to whieh a eomparison ean be made. 

3, Method 

a. Participants 

The partieipants in this study were primarily students of the Naval 
Postgraduate School (N=16): (3) female and (13) male, ranging from 21 to 43 years old. 
Within the group of 16, there were two Cadets from the Air Foree Aeademy and one 
from the U.S. Military Aeademy. The remaining partieipants were from three different 
aeademie eurrieula: 0399 (Modeling Virtual Environments and Simulation), 0365 
(Operations Researeh), and 0360 (Human Systems Integration). The HSI program is 
eonsidered non-teehnieal and the OR and MOVES degrees are technieal. The others 
were military officers ranging in pay grade from 01-05. Of these, there were offieers 
from the U.S. Navy, U.S. Army, U.S. Marine Corps, and the German Navy. Partieipants 
were randomly assigned to the two primary groups (TDG/TDS). These groups did not 
internet throughout the experiment. Additionally, the TDS and TDG groups were 
subdivided into two smaller internal groups (TDS Group A & B, TDG Group A & B). 
We eolleeted other demographie information: age; military oeeupational speeialty; 
eomputer, gaming, and simulation experienee; total serviee time; familiarity with infantry 
tactics, techniques, and proeedures; serviee eomponent; and eountry of eitizenship. The 
following tables provide a more eoneise view of our partieipants’ demographies. Table 4 
is granular and provides information by subjeet number on rank, age, time in serviee in 
years, and Computer Experienee in hours/year. 
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Table 4. Individual Summary Demographics 


General Demographics 

Subject # 

Rank 

Age 

TIS Years 

Computer 

Experience 

TDG 

8 

03 

27 

5 

2920 

746 

03 

32 

4 

1460 

547 

Cadet 

22 

3 

3650 

261 

04 

36 

14 

1095 

432 

04 

37 

14.33 

365 

663 

Cadet 

21 

3 

1825 

517 

Cadet 

21 

3 

1460 

97 

02 

26 

4 

4380 

TDS 

733 

04 

34 

11 

1095 

377 

05 

43 

22.92 

2190 

343 

01 

23 

0.92 

1825 

64 

03 

37 

14.67 

1095 

967 

03 

30 

7.92 

2190 

784 

03 

31 

8 

365 

13 

03 

32 

8.58 

730 

537 

03 

35 

15.92 

1825 


In addition to these individual demographics, Tables 5, 6, & 7 summarize 
the participants by TDG and TDS Group. 

Tables. Group Summary Demographics 



Age 

Time In Service 

Use of Personal Computer 


TDG Group 
(Years) 

TDS Group 
(Years) 

TDG Group 
(Years) 

TDS Group 
(Years) 

TDG Group 
(Hours/Year) 

TDS Group 
(Hours/Year) 

N 

8 

8 

8 

8 

8 

8 

Min 

21 

23 

3 

0.92 

365 

365 

Mean 

27.75 

33.13 

6.29 

11.24 

2144.38 

1414.38 

Median 

26.5 

33 

4 

9.79 

1642.5 

1460 

Max 

37 

43 

14.33 

22.92 

4380 

2190 

Std Dev 

6.54 

5.79 

4.91 

6.6 

1371.79 

688.06 


We can see in Table 5 that the computer use among the TDG group was 
higher than the TDS group. The TDS group had more time in service. This, also, 
indicates that they were, on average, older than the TDG group. 
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Table 6. Participants with Prior Commercial Video Game Experience 


1 INFORMATION ASKED | 

TDS 1 

1 TDG 1 

1 ALL 1 

Number of subjects having prior experience playing 
commercial video games 

7 

8 

15 

Number of subjects NOT having prior experience 
playing commercial video games 

0 

0 

0 

Number of people who did not respond to this question 

1 

0 

1 


We can see in Table 6 that every participant has had some level of 
experience playing commercial video games. 

Table 7. Summary of Commercial Video Game Experience 


Subjects Havin* Prior Ex0erience Pla^inK Commercial Video Games 

Tnw of Game 

TDS 1 

TDG 

ALL 

First person shooter 




Subjects Responding 

5 

4 

9 

Total # of Hours/Year 

132 

76 

208 

Minimum 

2 

2 

2 

Maximum 

96 

52 

96 

Mean 

26.40 

19 

22.70 

Standard Deviation 

39.63 

22.42 

31.45 

Flight Simulation 




Subjects Responding 

1 

1 

2 

Total # of Hours/Year 

96 

1 

97 

Minimum 

96 

1 

1 

Maximum 

96 

1 

96 

Mean 

96 

1 

48.50 

Standard Deviation 

0 

0 

67.18 

Racing 




Subjects Responding 

3 

0 

3 

Total # of Hours/Year 

23 

0 

23 

Minimum 

1 

0 

1 

Maximum 

20 

0 

20 

Mean 

7.67 

0 

7.67 

Standard Deviation 

10.69 

0 

10.69 

Sports (Football, etc.) 




Subjects Responding 

5 

2 

7 

Total # of Hours/Year 

367 

27 

394 

Minimum 

2 

3 

2 

Maximum 

240 

24 

240 

Mean 

73.40 

13.50 

43.45 
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Subjects Havin* Prior ExMrience Plains Commercial Video Games 

Tree of Game 

TDS 

TDG 

ALL 

Standard Deviation 

100.60 

14.85 

87.40 

Puzzle, Card, Board 




Subjects Responding 

5 

3 

8 

Total # of Hours/Year 

805 

774 

1579 

Minimum 

3 

2 

2 

Maximum 

730 

760 

760 

Mean 

161 

258 

209.50 

Standard Deviation 

318.55 

434.77 

338.40 

Strategy 




Subjects Responding 

3 

1 

4 

Total # of Hours/Year 

200 

12 

212 

Minimum 

24 

12 

12 

Maximum 

104 

12 

104 

Mean 

66.67 

12 

39.34 

Standard Deviation 

40.27 

0 

42.76 

Adventure, Fantasy 




Subjects Responding 

3 

3 

6 

Total # of Hours/Year 

41 

1119 

1160 

Minimum 

5 

12 

5 

Maximum 

24 

1095 

1095 

Mean 

13.67 

373 

193.34 

Standard Deviation 

9.61 

625.27 

441.77 

Arcade 




Subjects Responding 

2 

0 

2 

Total # of Hours/Year 

97 

0 

97 

Minimum 

1 

0 

0 

Maximum 

96 

0 

96 

Mean 

48.50 

0 

48.50 

Standard Deviation 

67.18 

0 

67.18 

Other 




Subjects Responding 

0 

1 

1 

Total # of Hours/Year 

0 

1 

1 

Minimum 

0 

1 

0 

Maximum 

0 

1 

1 

Mean 

0 

1 

0.5 

Standard Deviation 

0 

0 

0 


Table 7 shows that participants had the most commercial game experience 
with puzzle, card, board, and adventure/fantasy type games. Participants demonstrated 
the least experience with the racing type games. 


51 





































Participants were all treated in aceordance with ethical standards 
established by the American Psychological Association. Our study design was, also, 
approved by the IRB committee at NPS. 

We performed the role of instruetors by providing the introductory 
training for the study, including the infantry TTP training in Session 1. For Sessions 2-5, 
we took notes on performance, answered questions, and administered the surveys before 
and after each session. Additionally, we found, early in session 1 that managing a study 
of this size between two separate rooms was quite difficult for only two people. Thus, we 
enlisted the assistance of a U.S. Marine Corps student in the Operations Research 
curriculum to help with the study’s administrative tasks. 

b. Apparatus 

The lab space was purposefully configured in a way to support the 
isolation of the two internal TDG and TDS subgroups. Each subgroup occupied either 
Room A or Room B as seen in Figure 6. 



Figure 6. Experiment 2 - Fab Setup 

Each squad leader was provided with a Dell Pentium-4M laptop computer 
(6 all together) with CCM installed on it. The squad leaders used it to execute their own 
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orders, as well as, to faeilitate the eommands of the platoon leader. These laptops were 
part of a elosed experimentation network. This network was the basis for the CCM 
virtual environment that allowed six users (three offense and three defense) to internet 
with eaeh other relative to the CCM seenario for that session. The environment 
eomputers were eonneeted with a 16-port Ethernet Switeh using Cat 5 UTP eabling. 

For administrative purposes and training, an IBM laptop and an overhead 
projeetor were used to demonstrate infantry TTPs and the proeedures for partieipating in 
both the TDG and TDS sessions. We used two digital eameorders (audio/visual), one in 
eaeh room, to reeord partieipant aetivities during the sessions. We had multiple eopies of 
a Close Combat Marines Memory Sheet (see Appendix D), the operations order relevant 
to the partieular seenario (see Appendix K), and memory sheet for the offensive and 
defensive infantry taeties (see Appendix L) presented in the first session. There were 
white boards in both rooms for the subgroups to diagram their plans and briefs during 
their group discussion time. We provided scratch paper and pens on all tables for similar 
planning purposes. We used a twelve-sided die to randomly assign the participants to the 
groups. This die was, also, used to assign a unique identifier to each participant. 

c. Procedure 

From within the NFS student population, we acquired a total of 16 
participants. Then, we divided them into either a Tactical Decision-Making Game (TDG) 
group or a Tactical Decision-Making Simulation (TDS) group. The TDS group used 
Close Combat Marines (CCM). The TDG and TDS groups were further divided into two 
subgroups containing four subjects each. The TDG and TDS groups participated in the 
study at different times and on different days. See Table 8. 


53 



Table 8. Day and Time for TDG & TDS Sessions 


Week 1 


Week! 



^Thursday 



^nndiSI 

iJlonday 

Tnesday 

Wednesday 

0800-0900 








0900-1000 








1000-1100 








1100-1200 








1200-1300 


TDS 






1300-1400 


Sessions 3 & 4 



TDG 


TDG 

1400-1500 


TDG 



Sessions 3 & 4 


Session 5 

1500-1600 

TDS 

Sessions 1 & 2 




TDS 


1600-1700 

Session 1 & 2 





Session 5 



We chose different CCM battle scenarios in the different TDG/TDS 
sessions. To ensure a strong basis for comparison, our intent was to ensure that both 
groups received an equal number of scenarios. However, the TDS group mandated that 
one subgroup be in the offense while the other was in the defense. This was because the 
teams were fighting against one another in the virtual environment. On the other hand, 
the TDG group had no virtual environment. Thus, we controlled the nature of the 
scenario across each session (both subgroups were either on offense or defense). For 
example, the TDG Session 2 was defensive for both TDG subgroups and Session 3 was 
offensive for both TDG subgroups. The operation order and overall scenarios used in the 
TDG were drawn from the TDS CCM. 


(1) Tactical Decision-Making Game. The five TDG sessions 
consisted of one introductory session and four training sessions. In the first session, all 
TDG participants were provided with instruction on basic infantry offensive and 
defensive tactics, techniques, and procedures. We explained to them the exact nature of 
the TDG, how to participate in one, and the exact planning sequence they needed follow 
to complete the TDG effectively. The memory sheets for both the TTPs and the 
suggested planning process were available to participants and were distributed onto the 
tables for both groups. 

At the beginning of each TDG session (Sessions 2-5), we read the 
operation order to the entire cohort in Room B (see Figure 6). The groups were divided 
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into Group 1 and 2. Group 1 remained in Room B, while Group 2 moved to Room A for 
planning time. Each group discussed the operations order internally and, then, turned to 
individual planning. Each individual planned his response to the operation order and 
formulated his best effort at a course of action. After this phase, the individuals came 
back together for a group planning session. During this time, a leader was randomly 
selected to direct his team and to establish the overall team plan. Once each TDG 
subgroup had its team’s plan completed. Group 2 moved from Room B to Room A. 
Thus, the entire TDG group (8 persons in all) gathered in one room for an After Action 
Review (AAR). Each team leader, in turn, briefed his team’s plan to the entire TDG 
group. See Eigure 7. 



Eigure 7. TDG Scenario Elnderway 

The instructors and all participants were free to comment on and 


ask questions regarding the nature of the plans. This applied to why the team decided on 
a particular course of action. After this phase concluded, each TDG participant filled out 
a questionnaire related to the session that he had just finished. Additionally, the 
instructors evaluated the platoon leader’s leadership and decision-making performance on 
a separate evaluation form. The same process was repeated for the remaining three TDG 
sessions. 
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(2) Tactical Decision-Making Simulation. Upon arriving at the 
lab, the partieipants were divided into two TDS subgroups. The initial training session 
was the same for the TDS group as for the TDG group exeept for the hands-on CCM 
training. The two subgroups (eaeh had four partieipants) played against eaeh other in the 
remaining Sessions 2-5. This mandated that one group be in the offensive role and the 
other group in the defensive. Sinee the operation orders were different for eaeh group 
and knowledge about the eaeh other’s orders eould have potentially eompromised eritieal 
intelligenee, the subgroups were immediately separated so they would not be able to hear 
eaeh other’s internal diseussions. The subgroups remained separated in different rooms 
throughout the entire session (See Figure 6. Lab Spaee Layout). This teehnique ensured 
that eaeh team operated independently and did not have prior knowledge of their 
opponent’s plan. 

A platoon leader for eaeh TDS subgroup was randomly assigned at 
the beginning of eaeh session. That person aeted as the leader for not only the planning 
session, but for the eonduet of the entire TDS seenario for his subgroup. The groups 
were given approximately 3-5 minutes of planning time before they aetually started 
CCM. Onee CCM had started, the groups had a short time (2 minutes) to plaee their 
troops in the battle spaee. At this time, we started the aetual seenario where eaeh team 
began to implement their platoon leader’s plan within the virtual environment. Eaeh 
subgroup eonsisted of three squad leaders sitting at eomputer terminals while the platoon 
leader stood behind them, issued orders, and direeted the overall team aetion. The 
internal CCM system algorithms determined when to end the session based on the 
number of vietory loeations that eaeh group eontrolled and their number of troop losses. 
In all sessions, the seenarios were allowed to go to eompletion. 
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Figure 8. IDS Scenarios Underway 


We monitored the activities of the TDS subgroups seen in Figure 
8. One researcher was with Group 1 in Room A and the other was with Group 2 in Room 
B. Additionally, we acquired the assistance of an U.S. Marine Corps infantry officer. 
This person assisted us with the administration of the experiment, advice on infantry 
TTPs, and the discussion in the after action review (AAR) session for both the TDG and 
the TDS. (A detailed minute-by-minute schedule of the TDG and TDS sessions can be 
found in Appendix J). 


After each TDS session, each participant filled out a questionnaire 
and entered his/her opinions regarding how their teammates performed on decision¬ 
making, situational awareness, and leadership. We evaluated the platoon leaders on 
decision-making and leadership. Additionally, we used GRADE (Shattuck and Miller, 
2007) to evaluate each participant’s situational awareness at two points during each 
session (2-5). However, we were not able to accomplish this as we intended. There were 
certain days where TDS sessions ended earlier than expected and prior to the full 40 
minutes allotted. In cases where this occurred, we had only one GRADE worksheet (see 
Appendix M) from the participants to evaluate their SA for that session. 


4, Results 

This section lists results from two different types of analyses from Experiment 2. 
The longitudinal analysis was descriptive and examined each participant’s scores on 
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leadership, decision-making, and SA across all sessions. The inferential analysis 
conducted in sections b, c, and d. examined correlations among various factors of 
interest. 


a. Longitudinal Analysis over All TDG/TDS Sessions 

For the purpose of preserving our participants’ anonymity, we assigned 
them random subject identifiers which can be seen in all of the longitudinal graphs. We 
randomized their assignment to groups and their participant numbers to prevent one 
participant thinking he was better than another participant was, i.e., that a person with the 
number 1 would be ranked higher than a person with the number 2. 

The TDS leaders were appointed early in the scenario while the TDG 
leaders were appointed in the middle of the session only after individual planning had 
been completed. For a detailed minute-by-minute list of the TDG and TDS scenarios, see 
Appendix J - TDG/TDS Schedule (Experiment 2). 

(1) Leadership. For the leadership evaluations, each 
participant graded his entire team as a percentage of 100 (the total 100 points was divided 
among all team members). This grading method was present for only the leadership 
scores. The other dependent variables (decision-making and situational awareness) were 
graded on a Likert scale 1-7. The leadership data for the TDG and TDS participants 
revealed that the designated leader emerged as the perceived leader among all other 
participants without exception. 

Ligure 9 summarizes the results from the leadership evaluation 
between the TDG and TDS groups. 
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TDG Team A - Leadership 


TDG Team B - Leadership 




2 3 4 5 


967 784 -A— 13 537 


TDS Team A - Leadership 


TDS Team A - Leadership 


Figure 9. Longitudinal Results - Leadership (Self-Reported) 


Figure 9 shows that each TDS participant’s leadership scores 
fluctuated greatly between the sessions in which he/she acted as platoon leader versus 
those as a squad leader. The TDS scores had a much higher deviation from the mean 
than the TDG group. This could be attributed to the more active role played by the leader 
in the TDS group. The TDS platoon leader was generally standing behind his squad 
leaders directing their action only when they were actively playing CCM. No such 


59 










































































































activity was present in the TDG due to the nature of the TDG. In the TDS, though, the 
leaders were required to act, give orders, make ehanges to their plan, and suffered 
eonsequenees of inaction which resulted in troop losses and attaeks by a thinking enemy. 
These activities appeared to propel the TDS leader to a higher level of performanee — a 
faet born out by the results of the overall seoring by his subordinate squad leaders. 

(2) Decision-Making. The overall decision-making trend for 
the TDG group was relatively flat as seen in Figure 10. The TDS teams were either 
overall inereasing for Team A or deereasing for Team B. Partieipant number 343 in the 
TDS Team A was absent from Sessions 3 and 4 and, therefore, reeeived no seore (a seore 
of zero). Interestingly, this trend of teammates grading one another higher in deeision- 
making, when they won the seenario, and lower, when they lost, eorresponded to the 
win/loss results and spanned all sessions for TDS teams. Team A was defeated 
deeisively in Sessions 2, 3, and 4, but made improvements in their ability to work 
together and in their deeision-making ability. Team B’s performanee deereased over the 
eourse of all sessions. Even though they won all but the last session. Team A eontinued 
to improve at a faster rate than Team B. They were defeated in the last session due to the 
superior firepower (availability of artillery) used by their opponent. These trends eould 
indieate that the team members were equating their ability to make deeisions with the 
outeome of the sessions. 

Even though all teams were randomly assigned, three of the four 
members of TDS Team B had been classmates in previous NPS eourses and are now all 
in the same Human Systems Integration eurrieulum. The graph of TDS Team B eould 
indieate an initial overeonfidenee in their ability to work together. However, this 
deereased over time as the other team beeame more sueeessful. The graph of TDS Team 
A - Deeision-Making eould indieate that the group was eoaleseing as a team throughout 
all sessions. They worked better together and were able to make better deeisions as the 
sessions progressed. 

Eigures 10 and 11 demonstrate a eonsisteney between the sessions 
on deeision-making and situational awareness. As stated earlier, partieipant 343 of TDS 
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Team A received a score of zero in Sessions 3 and 4. This was a result of 343’s absence 
from the experiment during those two sessions. 



2 3 4 5 


-•—8 -■—746 -A— 547 261 


TDG Team A - Decision-Making 



Session Session Session Session 
2 3 4 5 


733 -■— 377 -A— 343 64 



TDG Team B - Decision-Making 



2 3 4 5 


967 -■— 784 -A— 13 537 


TDS Team A - Decision-Making 


TDS Team B - Decision-Making 


Figure 10. Longitudinal Results - Decision-Making (Self-Reported) 


(3) Situational Awareness. The graphs in Figures 10 and 11 are 
almost identical for both TDG and TDS groups. The participants’ scores indicate that the 
TDG and TDS groups’ situational awareness scores over all sessions correspond strongly 
to their decision-making scores. 
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Figure 11 provides a basis for eomparing SA between the two 
groups (TDG & IDS). 



2 3 4 5 


^8 -■—746 —A— 547 261 


TDG Team A - Situational Awareness 



TDG Team B - Situational Awareness 



TDS Team A - Situational Awareness 



TDS Team B - Situational Awareness 


Figure 11. Longitudinal Results - Situational Awareness (Self-Reported) 


The previous longitudinal graphs show the overall results for 
leadership, decision-making, and situational awareness for all participants in the TDG 
and TDS groups across all sessions. 
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During this study, we were eoneemed with several more 
parameters beside the leadership, deeision-making, and situational awareness skills. We 
were, also, interested in how other key skills required of a platoon level leader eould be 
improved through the use of the TDG and TDS. Figures 12 and 13 show the absolute and 
relative improvement within the two training groups (TDG and TDS) from the beginning 
session to the final session. We drew this information from questions 4 and 5 in the final 
TDG questionnaire and questions 14 and 15 in the final TDS questionnaire. The scores 
reflected in Figures 12 and 13 are averages for all participants in the TDG and in the TDS 
groups. They, also, show that there was a perceived improvement across all domains. 
Figures 12 and 13 demonstrate the difference between the absolute scores at the 
beginning of Session 1 and the absolute and relative improvement reported in the final 
questionnaire after the final session. 



Y 

+ Initial Absolute Score 
X Final Session Relative Improvement 
□ Final Session Absolute Score 


Figure 12. TDG Absolute and Relative Improvement 
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In Figure 12, we can see that participants stated that they showed 
improvement over all domains because each line has some measurable length. Since the 
TDG is a pen and paper drill with no simulation involved, participants only had the 
ability to engage in the scenario within their own mind. They never saw it played out in 
an objective manner. We believe that this was the reason why the “Tactical Movement to 
an Objective” and the “Fire & Maneuver” tasks showed the least amount of 
improvement. However, overall perception of absolute and relative improvement across 
all domains was small. Although TDG and TDS subjects participated equally in 
offensive and defensive scenarios, the TDG group believed that they improved more on 
the defensive tasks. In Figure 13, we can see that the magnitude of the improvements in 
the TDS group was greater than in the TDG group. However, this was not by a 
significant margin. 



Y 

+ Initial Absolute Score 
X Final Session Relative Improvement 
□ Final Session Absolute Score 


Domain 


Figure 13. TDS Absolute and Relative Improvement 
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In Figure 13, TDS participants stated that they felt they improved 
over all domains, though some more than others. We noticed that in the “Issue Order” 
domain, their relative improvement was far less than their absolute score. The domains 
“Tactical Movement in the Offense” and “Tactical Movement Objective”, also, 
demonstrated this same condition, but was by an insignificant amount. We believe that 
the minimal improvement on the “Issue an Operation Order” task was because 
participants did not issue operation orders; rather, the orders were only provided for them 
in written form. The TDS participants, also, practiced offensive and defensive tactics 
equally. However, they thought that they improved most in offensive operations. This is 
in direct opposition to the TDG group results. We believe that the TDG format implied, 
to the students, that they were in more of a defensive role, while the TDS format implied 
more of an offensive role. 

b. Hypothesis Testing - TDG/TDS Participant Scores 

Since our experiment was about testing and evaluating leadership, 
decision-making, and situational awareness, we decided to examine them based on 
differences between the TDG and the TDS groups. A t-test was used because there were 
only two different experimental groups. Since there was no good reason to believe that 
the scores would fall in a particular direction, we decided to use a 2-tailed t-test to 
evaluate the variables. For all hypothesis tests, we chose the a level = 0.05 to be 
significance level controlling whether to reject the null hypothesis or not. 

(1) Peer Evaluation. 

Leadership . The participants’ peer leadership scores were taken 
from question 2 on questionnaires from Sessions 2-5. Each participant received a 
leadership score from his teammates such that there were a total of three different scores 
per participant. This was averaged to achieve a session average and was repeated for all 
sessions. Einally, all participant session averages were averaged again to find the total 
leadership score for each participant. 

Ho (Null Hypothesis) = There is no difference between the TDG 
and TDS participant leadership scores given by their peers. 
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Ha (Research Hypothesis) = There is a difference between the 
TDG and TDS participant leadership scores given by their peers. 

A 2-tailed t-test determined p = 0.8747 > 0.05. We do not reject 
the null hypothesis. We conclude that there is no difference between the participants’ 
scores when evaluated by their peers on leadership when comparing the TDG and TDS 
groups. This suggests that the TDG and the TDS provide opportunity to exercise and to 
evaluate leadership. 

Decision-Making . The participants’ peer decision-making scores 
were taken from questions 8, 9, and 12 on questionnaires from sessions 2-5. Each 
participant received a decision-making score from his teammates such that there were a 
total of three different scores per participant. This was averaged to achieve a session 
average and was repeated for all sessions. Finally, all participant session averages were 
averaged again to find the total decision-making score for each participant. 

Ho (Null Hypothesis) = There is no difference between the TDG 
and TDS participant decision-making scores given by their peers. 

Ha (Research Hypothesis) = There is a difference between the 
TDG and TDS participant decision-making scores given by their peers. 

A 2-tailed t-test determined p = 0.0356 < 0.05. We do not reject 
the null hypothesis. We conclude that there is a difference between the TDG/TDS 
participants’ scores on the peer decision-making evaluation. Since we assigned the 
participants into groups randomly, we think that the real difference was caused by the 
method used, not by differences in how the peers scored each other within the groups. 

Situational Awareness . The participants’ peer Situational 
Awareness scores were taken from questions 12 and 16-18 on questionnaires from 
Sessions 2-5. Each participant received a situational awareness score from his teammates 
such that there were a total of three different scores per participant. These were averaged 
to achieve a session average and were repeated for all sessions. Finally, all participant 
session averages were averaged again to find the total decision-making score for each 
participant. 
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Ho (Null Hypothesis) = There is no differenee between the TDG 
and TDS partieipant Situational Awareness seores given by their peers. 

Ha (Researeh Hypothesis) = There is a differenee between the 
TDG and TDS partieipant Situational Awareness seores given by their peers. 

A 2-tailed t-test determined p = 0.0068 < 0.05. We rejeet the null 
hypothesis and eonelude that there is a differenee between the partieipants’ seores on the 
peer situational awareness evaluation in TDG and TDS. Sinee we assigned the 
partieipants into groups randomly, we think that the real differenee is eaused by the 
method used and not by differenees in how the peers in the groups seored one another. 

(2) Researeher Evaluation. 

Leadership. We, also, evaluated the TDG and TDS partieipants 
with respeet to their leadership. Their seores were taken from the Researeher Evaluation 
Eorm (see Appendix N) questions 15-16. Eaeh partieipant was ehosen to be the team 
leader for exaetly one session. This was the only session for whieh we evaluated him and 
obtained a leadership seore. 

Ho = (Null Hypothesis) There is no differenee between TDG and 
TDS partieipant leadership seores given by the researehers. 

Ha = (Researeh Hypothesis) There is a differenee between TDG 
and TDS partieipant leadership seores given by the researehers. 

A 2-tailed t-test determined p = 0.1765 > 0.05. We do not rejeet 
the null hypothesis. We eonelude that there is no differenee between the partieipants’ 
seores on researeher leadership evaluation in TDG and TDS. 

Deeision-Making. We also evaluated the TDG and TDS 
partieipants with respeet to their deeision-making ability. Their seores were taken from 
the Researeher Evaluation Eorm (see Appendix N) questions 13-14. Eaeh partieipant was 
ehosen to be the team leader for exaetly one session. This is the only session for whieh 
we evaluated him and obtained a leadership seore. 


67 



Ho = (Null Hypothesis) There is no differenee between the TDG 
and TDS partieipant decision-making scores given by the researchers. 

Ha = (Research Hypothesis) There is a difference between the 
TDG and TDS participant decision-making scores given by the researchers. 

A 2-tailed t-test determined p = 0.1606 > 0.05. We do not reject 
the null hypothesis. We conclude that there is no difference between the participants’ 
scores on the researcher decision-making evaluation in TDG and TDS. 

Situational Awareness. We were able to evaluate the TDS 
participants on their SA, but not for the TDG participants. The TDS group had a CCM, 
which maintained a state of all enemy and friendly unit locations. As we mentioned 
earlier, we drew their ground truth from this system with an SA instrument called the 
Geographical Recall and Analysis of Data in the Environment (GRADE) (Miller & 
Shattuck, 2007). We compared each participant’s GRADE sheet with ground truth on a 
Eikert scale 1-7. Their average SA score was a 4.19. The TDG group had no ground 
truth from which to draw because they did not use a computer system that maintained a 
record of their planning process. As a result, their SA could only be evaluated based on a 
subjective peer assessment from the questionnaires. 

(3) Summary of Results. Results of the Hypothesis tests for 
leadership, DM, and SA are located in the tables below. Table 9 presents the summary of 
the peer evaluations and Table 10 presents the instructors’ evaluations. 


Table 9. Peer Evaluation t-test Comparing TDG/TDS 



P Value 

Result 1 

Leadership 

p = 0.8747 >0.05 

No significant difference between 
TDG/TDS leadership scores 

Decision Making 

p = 0.0356 <0.05 

Significant difference between 
TDG/TDS decision-making scores 

Situational Awareness 

p = 0.0068 < 0.05 

Significant difference between 
TDG/TDS SA scores 
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Table 10. Instructor Evaluation t-test Comparing TDG/TDS 



P Value 1 

Result 1 

Leadership 

p = 0.1765 > 0.05 

No significant difference between 
TDG/TDS leadership scores 

Decision Making 

p = 0.1606 >0.05 

No significant difference between 
TDG/TDS leadership scores 

Situational Awareness 


* No Basis for Comparison * 


The TDG was a mental simulation that was not based upon a 


collective understanding of ground truth for the scenario. Each person had his own 
understanding of where the enemy was and what they were doing. The TDS group had 
ground truth established from within the computer simulation. Due to the varying nature 
of these two methods for determining situational awareness, there was no basis from 
which to compare the TDG and the TDS from the instructor’s point of view. 

c. TDG Correlation Analysis 

All factors were scored based on each participant’s subjective perception 
of the questions asked. In other words, if the participant was grading his peers on 
leadership, he scored them based on his own opinions and understanding of what a good 
or bad leader should be. This was true for all factors in the below analysis. 

(1) Eactor: Talkativeness Predicts Eeadership. 

Questions 

• Eeadership Question (Sessions 2-5/Question 2) - Each 
participant was asked to provide a score between 0 and 100 
for each of their team members. The final score must total 
100. This score essentially ranked the participants with 
regard to their perceived leadership ability during the 
sessions. 

• Talkativeness Question (Sessions 2-5/Question 6) - The 
same scoring method (between 0 and 100) was used for the 
amount of talkativeness exhibited by the participants during 
the sessions. 
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Results 


Figure 14 demonstrates that there is a strong linear relationship 
between talkativeness and the perception of leadership. 



Regression Plot 



Actual by Predicted Plot 


Figure 14. Graph - Talkativeness/Leadership 


The P value for the test for significance of regression is P = 0.0005 
and RSqr = 0.88. This clearly indicates strong evidence that talkativeness is linearly 
related to leadership. 


(2) Factor; Age Predicts Leadership. 

Questions 

• Age Question. (Session 1/Question 9) - This question 
asked for the participant’s year of birth. The age was 
determined by subtracting this from the current year. 

• Leadership Question. (Sessions 2-5/Question 2) - see 
Factor: Talkativeness Predicts Leadership. 

Results 


Figure 15 demonstrates a strong correlation between age and the 
perception of leadership. 
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Regression Plot 



Actual and Predicted Plot 


Figure 15. Graph - Age/Leadership 


The P value for the test for significance of regression is P = 0.0262 
and RSqr = 0.59. The regression Model indicates that Age is a strong predictor of 
perceived leadership ability. In other words, the older the participant was the stronger 
leadership score he received. 


Making. 


(3) Factor: Situational Awareness Predicts Good Decision- 


Questions 

• Decision-Making Question (Sessions 2-5/Question 12: 
Items 8 & 9) - Participants were asked about their opinion 
of their teammates with regard to their ability to identify 
the enemy’s center of gravity and to attack it. Also, they 
were asked about how well the other team members 
adjusted to the changing enemy situation. All scores were 
graded on a 1-7 Likert Scale. 

• Situational Awareness Question. (Sessions 2-5/Question 
12, Items 16, 17, & 18) - Participants were asked about 
their confidence regarding the enemy location, enemy size, 
and direction of enemy movement. 

Results 


Figure 16, also, demonstrates that SA is strongly correlated with 
the ability to make decision-making. 
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Regression Plot 


Actual by Predicted Plot 


Figure 16. Graph - Situational Awareness/Decision Making 


The P value for the test for significance of regression is P = 0.0004 
and RSqr = 0.89. This clearly indicates strong evidence that decision-making is linearly 
related to situational awareness. 

(4) Factor; Correlation between the Internal Team Members’ 
Scores and the Instructors’ Score on Leadership. 

Questions 

See above Talkativeness Predicts Leadership. This measure 
examined the relationship between the leadership score a participant received from his 
fellow teammates and the leadership score the instructors gave him during his role as 
team leader. 

Results 

Figure 17 shows that there is some relationship between the 
instructor leadership score and the leadership score that the participants received from 
their peers. 
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Regression Plot 



Actual by Predieted Plot 


Figure 17. Graph - Instructor Leadership/Perceived Leadership 


The P value for the test for significance of regression is P = 0.0143 
and RSqr = 0.66. This clearly indicates that there is evidence that the instructor 
evaluation of the participants’ leadership ability is linearly related to the leadership 
evaluation given to the participant by his peers. 

d. TDS Correlation Analysis 

(1) Factor; Talkativeness Predicts Leadership. 

Questions 

• Leadership Question (Sessions 2-5/Question 2) - Each 
participant was asked to provide a score between 0 and 100 
for each of their team members. The final score must total 
100. This score essentially ranked the participants with 
regard to their perceived leadership ability during the 
sessions. 

• Talkativeness Question (Sessions 2-5/Question 6) - The 
same scoring method (between 0 and 100) was used for the 
amount of talkativeness exhibited by the participants during 
the sessions. 

Results 

Figure 18 demonstrates a weak correlation between talkativeness 

and leadership. 
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Regression Plot 


Aetual by Predieted Plot 


Figure 18. Graph - Talkativeness/Leadership 


The P value for the test for signifioanee of regression is P = 0.005 
and RSqr = 0.48. This indieates evidenee that talkativeness is linearly related to 
leadership. 


(2) Faetor; Age Prediets Leadership. 

Questions 

• Age Question. (Session 1/Question 9) - This question 
simply asked for the partieipant’s year of birth. The age 
was determined by subtraeting this from the eurrent year. 

• Leadership Question. (Sessions 2-5/Question 2) - see 
Faetor; Talkativeness Prediets Leadership. 

Results 


The regression plot below in Figure 19 demonstrates that age is 
eorrelated with the pereeived leadership seores. 
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Regression Plot 


Actual By Predicted Plot 


Figure 19. Graph - Age/Leadership 


The P value for the test for significance of regression is P = 0.028 
and RSqr = 0.58. This indicates evidence that age is linearly related to leadership. 


Making. 


(3) Factor; Situational Awareness Predicts Good Decision- 


Questions 

• Decision-Making Question (Sessions 2-5/Question 12; 
Items 8 & 9) - Participants were asked about their opinion 
of their teammates with regard to their ability to identify 
the enemy’s center of gravity and to attack it. Also, they 
were asked about how well the other team members 
adjusted to the changing enemy situation. All scores were 
graded on a 1-7 Likert Scale. 

• Situational Awareness Question. (Sessions 2-5/Question 
12, Items 16, 17, & 18) - Participants were asked about 
their confidence regarding the enemy location, enemy size, 
and direction of enemy movement. 

Results 


Figure 20 clearly demonstrates the strong correlation between 
decision-making and situational awareness. 
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Figure 20. Graph - Deeision-Making/Situational Awareness 





Regression Plot 


The P value for the test for signifieanee of regression is P < 0.0001 
and RSqr = 0.97. This elearly indieates strong evidenee that deeision-making is linearly 
related to situational awareness. 

(4) Correlation between Time in Serviee (TIS) and Leadership. 

Questions 

• Time in Service Question; Session 1/Question 15 - This 
question asked for the participant to state the number of 
years and months he has been in military serviee. 

• Leadership Question; (Sessions 2-5/Question 2). 

Results 

Figure 21 shows a strong eorrelation between TIS and leadership. 
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Figure 21. Graph - Time in Serviee/Leadership 



Residual Plot 



The P value for the test for signifioanee of regression is P = 0.0226 
and RSqr = 0.61. This indieates evidenee that TIS (months) is linearly related to 
leadership. 

e. TDG/TDS Summary of Results 

Table 11 summarizes the eorrelation results from Experiment 2. P Values 
below 0.05 indieate signifioanee. 

Table 11. Summary Comparison of TDG/TDS Results for Signifioanee 



TDG 

TDS 


P value 

R-Squared 

P value 

R-Squared 

Leadership (Peer Evaluated) 
VS Talkativeness (Peer 
Evaluated) 

0.0005 

0.88 

0.005 

0.48 

Strong Correlation 

Correlation 

Leadership (Peer Evaluated) 
VS Age 

0.0262 

0.59 

0.018 

0.058 

Correlation 

Correlation 

Leadership (Instructor 
Evaluated) VS Leadership 
(Peer Evaluated) 

0.0143 

0.66 

0.617 

0.04 

Correlation 

No Correlation 

Situational Awareness (Peer 
Evaluated) VS Decision 
Making (Peer Evaluated) 

0.0004 

0.89 

<0.001 

0.97 

Strong Correlation 

Strong Correlation 

Time in Service VS Leadership 
(Peer Evaluated) 

0.09 

0.4 

0.0226 

0.61 

Weak Correlation 

Correlation 


77 














































/. Qualitative Analysis 

The experiment produced a number of qualitative results that were 
interesting and should be included. These include data, such as verbal participant 
comments regarding their group’s experience, researcher observations, as well as, the 
participant’s answers to open-ended questions from the questionnaires. In some cases, 
these results were more telling than the quantitative ones. 

(1) Training Session Duration and Timing. The duration and 
the number of individual training sessions that are required to achieve a particular skill 
level have been largely unexplored. We conducted our experiment in five total sessions 
lasting one hour each. To begin the sessions, we read the operation order and turned the 
teams over to 3-5 minutes of planning time. After their planning time ended, they had 
another 3-5 minutes to get their troops into their starting positions before the actual 
scenarios started. Each actual simulation scenario lasted approximately 40 minutes. We 
knew that the 3-5 minutes of planning time was not sufficient to create a good tactical 
plan, but we were constrained by the scenarios lasting 40 minutes. The planning that the 
teams conducted was quite surprising. We observed that the TDS groups actually had 
planning time left over and were waiting to start the actual scenario. They appeared to be 
truncating their plan and rushing into the execution phase. This could be attributed to 
their heightened level of excitement when preparing to interact with the simulation. This 
was not only true at the outset of the CCM scenarios, but during the middle of the 
sessions, too. If a group had a plan to counter some enemy action, they seldom, if ever, 
executed it. There seemed to be an urge to forget about what they had planned and only 
use CCM on a moment-by-moment basis. 

(2) Overall Session Time was Too Short. The pauses in the 
sessions that were required to collect SA information ended up pushing the end of the 
simulation longer than expected. This left virtually no time for participants to fill out 
their questionnaires. At least 10 minutes should be allowed for participants to properly 
fdl out the questionnaires without feeling rushed. In some cases, the TDG group was 
able to complete their task in the allotted 1-hour time slot, but even they were rushed 

during some sessions. The time for each session should be extended to at least 1.5 hours 
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and preferably two hours. The latter would allow the participants to have sufficient time 
to plan, complete the entire TDG/TDS scenario, and fill out their questionnaires before 
departing the area. 

(3) Combining the Best of the TDG and the TDS. Military 
schools that plan to use this type of training should encourage the time tested and 
superior planning process from the TDG to take place at the beginning of the training 
sessions. Once unit leaders have established their plans, they should implement them in 
the TDS to see how each plan holds up under enemy contact. This, also, applies to how 
the unit leaders and subordinate leaders interact and respond in the tactical virtual 
environment. The TDS will provide feedback, and an ultimate outcome, which can be 
discussed in an after action review in terms of the original plan. Subject matter experts 
should be present throughout to prepare for a robust after action review session. 

(4) Team Cohesiveness. We noticed improvement in team 
cohesiveness over the three days and five separate sessions. During the first session, the 
TDS sub-groups were somewhat hesitant to communicate with one another before and 
during the battle. Each participant seemed to be focused on his particular task and paid 
little attention to his teammates. After the final session, there was a good deal of team 
cohesion among all groups. Particularly in the final TDS session, TDS Group 1 was 
victorious and burst out of their seats to slap hands and congratulate each other. 

(5) Confusion and Loss of Situational Awareness. Some TDS 
participants had trouble keeping track of which units were friendly and which were 
enemy. In their view, they did not know which squads or units were on their side. At 
times, their confusion led them to fire on their own adjacent units. This was obviously 
not an issue in the TDG group. This could be used as an important teaching point 
regarding the confusion of combat and how this could contribute to friendly fire 
incidents. Instructors could emphasize how important it is to maintain SA at all times. 

(6) Pleasant Training Experiences Contribute to Learning. Not 
all military training can be fun. To emphasize that there are times when the mission has 
priority even at the expense of one’s comfort, some military training must be unpleasant. 
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However, some training industry experts believe that a pleasurable training experience 
can contribute to participant learning. According to Michael Allen of Allen Interactions, 
a Minneapolis training firm, “People remember things through humor.” “When you talk 
about a lasting learning experience, you recall what is notable about the event. If 
something unusual or funny happens, you'll probably remember it” (Fister, 1999). 

During the final after action review and once all sessions had 
concluded, the IDS participants reported they were motivated and enjoyed the 
experience. Conversely, the TDG group reported they became bored with it. This 
boredom was a fact supported by some of the note pages we collected from the TDG 
group after their sessions. Some had extensive doodle marks on them indicating that this 
particular person was able to “wander off’ mentally from the planning task in which he 
was involved. In part of our final questionnaire, we asked participants in both TDG and 
TDS groups about the quality of their individual training experiences. The scores in 
Table 12 were taken from the final questionnaires and illustrate interesting differences 
between the two approaches. 


Table 12. Comparison of Quality of Training 


1 TiJG 

^1 Tds 

1 

Question 

Average Score 
(1-7 Likerf) 

Question 


"When you think about your future 
training needs, to what extent would you 
like it to be supplemented with TDGs?” 

2.75 

“When you think about your future 
training needs, to what extent would you 
like it to be supplemented with computer 
based simulations like CCM?” 

4.625 

“When you think about your future 
training needs, to what extent would you 
like it to be supplemented with computer 
based simulation of a TDG?” 

5 

* Question Not Applicable * 


"How well did the TDG sessions meet 
your expectations regarding training 
value?" 

3.1 

"How well did the TDS sessions meet 
your expectations regarding training 
value?" 

4.5 

"The TDG experience was able to 
provide me with a good mental picture of 
the battle space." 

3.3 

"Rate the simulation's ability to provide 
you with VISUAL cues so that you can 
identify all threats within the mission." 

■ 

"Rate the simulation ability to provide 
you with AUDIO cues so that you can 
identify all threats within the mission." 

4.6 

"In your opinion, what was the overall 
quality of training in the TDG?" 

3.2 

"In your opinion, what was the overall 
quality of training in the TDS?" 

4.9 
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Table 12 indicated a desire by all participants to incorporate 
computer based simulation into their training. What makes this so significant is that the 
TDG group indicated displeasure with their task and opted for an alternative to it in 
computer-based simulation. The TDS participants enjoyed their training enough to 
continue to recommend it be included in future tactical decision-making training 
regimens. The scores in Table 12, also, indicate that the perception of the quality of 
training was lower in the TDG group than in the TDS group. 

(7) Verbal and Non-Verbal Communications. Some of the 
participants’ communications (verbal and non-verbal), and interactions during the 
sessions, warrant comment. Our intent was to simulate the communications between 
platoon leaders and squad leaders in the most realistic way possible. In a field 
environment, a platoon leader usually traveled with one of his squads. This prevented 
him from seeing exactly what the other two squad leaders were observing at any 
particular point in time. We simulated this by advising participants before the sessions 
began that each squad leader should remain at his own computer and only look at his own 
screen. Likewise, the platoon leaders should remain with a single squad throughout and 
not hop back and forth between screens. Verbal communication would be heard by all as 
if they were all on the same radio network. This did not occur. During the sessions, we 
constantly reminded the squad leaders to remain in their seats and to keep their focus on 
their screens (their view of the battle space). However, they got up to help their fellow 
squad leaders who were having trouble or who were unable to understand what was going 
on in their view of the simulation. In like manner, the platoon leaders were consistently 
looking over at their other squad leader’s computer screens in, what we believe, was an 
attempt to garner more SA before they made decisions. 

5. Summary 

The following five points were significant findings from the summary of 
quantitative and qualitative results in Experiment 2. 

• In the longitudinal analysis across all sessions, we observed that 
participants received their highest leadership scores when they were 
designated as the leader in the TDG and the TDS sessions. However, the 
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TDS participants had a much higher deviation from the mean on 
leadership beeause they were aeting in the role of a leader by standing, 
issuing orders, and supervising their squad leaders during the virtual 
battle. 

• TDS partieipants seemed to equate a favorable outeome in the simulation 
with good deeision-making. This may or may not be true, but their seores 
eonsistently supported this eoneept. We found that peer evaluations were 
strongly eorrelated between the partieipants’ deeision-making and 
situational awareness seores. 

• We tested the hypothesis that there was a differenee between the TDG and 
TDS leadership, deeision-making, and SA seores. We believed that the 
methods were the eause of differenees with respeet to deeision-making 
and situational awareness when evaluated by peers. When we evaluated 
the groups, there was no signifieant differenee between the methods when 
examining deeision-making and leadership. We were not able to evaluate 
if there was a signifieant differenee between the groups on situational 
awareness beeause we were unable to obtain SA data from the TDG 
group. 

• Internal to the TDG and TDS groups, we found that pereeived leadership 
was eorrelated with talkativeness, time-in-serviee, and age. SA and 
deeision-making were strongly eorrelated in both groups. 

• Our qualitative assessment determined that an improved approaeh to 
training with taetieal deeision-making simulations should be eonsidered. 
We believe that a new approaeh, whieh we eall the Integrated Taeties and 
Planning Simulation Exereise (ITAPSE), would bridge the best of both 
training approaehes (TDS & TDG). The TDS was eondueted almost 
exelusively within the virtual environment and seemed to be more 
engaging to the partieipants. The TDS experienee was exeiting and 
allowed eaeh partieipant to play a highly aetive role. The TDG was 
eondueted with pen and paper. It prompted more thought about individual 
aetions and fostered more thorough planning. The ITAPSE eombines the 
planning teehniques of the TDG with the ability to implement and exeeute 
that plan via the TDS. 
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IV. RECOMMENDATIONS AND FOLLOW-UP EXPERIMENT 


A. RECOMMENDATIONS 

After the experiment and data analysis were completed, there were several 
recommendations that we identified. These fall into the following categories: 

1, Improvements to the CCM Software for TDS Use 

We believe that if CCM is to continue to be used as a TDS for training, the 
following modifications to the code and meta-data would provide for a richer experience 
for the students: 


a. Operation Orders 

Administrators should ensure that the operation orders are corrected so 
that the information is accurate and consistent for both groups. The orders used in this 
experiment contained numerous acronyms and jargon that was specific to an infantry 
environment. This would be acceptable for military personnel who have been trained in 
infantry TTPs, but not for members of other services and especially not for Foreign 
Service members. 

Some information that would be critical to the TDG group was excluded 
entirely from the orders. The orders were adapted for use by the TDG group, but were 
originally written by the CCM developers for the TDS group. Information, such as the 
current location of one’s own unit, was not included. The TDS group would immediately 
understand where their unit was located when they started CCM, but the TDG group had 
no ability to get this information. We had to estimate the TDG groups’ locations during 
the TDG sessions to give them a starting point. 

Orders, also, had conflicting and erroneous information. The operation 
orders that we used are located in Appendix K. Session 1-la’s Operation Order stated 
that the company had been provided no mortar support, yet the Task Section of the order 
stated plainly that mortar priority of fire was to third platoon. 
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b. 


Pause Feature 


When the current system was paused, it was difficult to establish ground 
truth of unit locations because each participant’s screen was on a different part of the 
battlefield. Internal simulation algorithms, that display only what the participant’s team 
avatars within the simulation are capable of seeing based on distances and terrain 
features, controlled the number of troops visible to the individual. 

c. Integrated Situational Awareness Measurement Tool (ISAM-T) 

Future editions of CCM should incorporate a built-in SA evaluation tool 
that will present an SA screen to the user at preset or instructor controlled points during 
the sessions. The GRADE SA evaluation tool used in this experiment was difficult to 
administer because the TDS did not have a good mechanism to pause the simulation. On 
two separate occasions, users accidentally quit the TDS and forced a complete restart of 
the scenario. This wasted approximately 20 minutes of session time. 

Additionally, ISAMT could internally calculate distances, size, and 
locations of friendly/enemy units and compare this to ground truth. This will return an 
objective difference score between where the student thought the enemy/friendly forces 
were and where they actually were. Based on this criteria, the simulation will grade the 
student and provide an objective measure of SA. This information could, also, be saved 
and used by instructors/SMEs in the AAR portion of the scenario. 

d. XML Output File 

“Many simulations provide none or only rudimentary support for AAR” 
(Sadagic, 2007). CCM does provide some after action information output to the screen at 
the end of the simulation, but here is no file on which a researcher could run statistics to 
look for possible trends. We suggest incorporating an XME output file that would 
provide results for each player in CCM. This could be used to extract objective data for 
each player in the team. It could provide a way to evaluate their tactical decision-making 
in terms of distances covered at a run, number of rounds expended per unit time, unit 
proximity to adjacent units, and whether or not the student fired on friendly troops. 
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e. Instructor/Administrative Workstation 

CCM should include an administrative workstation that would allow for 
instructor’s observation and control of friendly and enemy units. This would allow the 
instructor to maintain oversight of both sides during the entire scenario. From this 
station, he would be able to pause the TDS and administer the automated SA evaluation 
that we suggested as a new feature. 

/, True Record/Playback 

We reeommend ineorporating a true reeord/playbaek feature for aetion 
review purposes into this master workstation. The eurrent playbaek feature within CCM 
has some level of interaction available to it (the user ean interaet with the reeorded 
session as it is playing and alter the outeome). Additionally, in the eurrent CCM reeord 
feature, time is constant and non-variable (1 second = 1 second). We recommend that 
developers include a fast-forward capability so that TDS administrators can move quickly 
to critical points in the battle to illustrate specific learning. 

g. Virtual White Board 

We recommend including a virtual white board overlay. Instructors would 
draw on the master screen to illustrate tactical accomplishments or mistakes. The 
instructor screen drawing could show up on some or all students’ screens. 

h. Re-designation of Unit Leadership 

If the actual platoon leader is killed in battle, CCM should allow for the 
designation of another platoon leader. CCM has leadership rings both around each team 
and team leader to indicate that certain individuals have more leadership capability than 
others. The proximity of troops to their leader improves their morale. If a team leader 
dies in battle, his troops will be more likely to panie in the faee of enemy fire. 
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2 . 


Potential Bugs in CCM. 


a. Observer Mode 

We believe we identified a bug in the software. During one of the 
sessions, two players' sereens went into observer mode. This term “observer mode” 
deseribes a condition where visualization on screen was present, but interaction with the 
system was not possible for the participant. One reason for this resulted when a 
participant’s entire squad was killed by enemy fire and, therefore, had no troops to 
control. However, the other reason for the case of “observer mode” was undetermined. 
We looked through the manuals for indications as to why this would have happened, but 
found none. 


b. Potential Memory Leak in CCM 

We noticed some anomalies with the networked configuration of CCM 
during the TDS sessions. Future administrators of the experiment should ensure that 
computers are freshly booted before starting each session. On one occasion, to reduce the 
amount of setup time, we left the machines on overnight and at the network setup screen. 
After this long period of computer inactivity, some of the computers would not 
synchronize with the host in multiplayer mode. We finally had to reboot all six 
computers before CCM would synchronize over the network. 

B, SUGGESTIONS FOR A FOLLOW-UP EXPERIMENT 

If future research efforts try to replicate this study, or to examine certain 
components in more detail, then researchers might want to consider the following; 

1. Participants 

Search for a pool of subjects that are closer to the actual population for which 
TDS training will be implemented. This might be used to confirm the previous findings 
in this study. The follow-on study should be conducted at a site where a traditional 
training approach to tactical decision-making is already being used so that the training 
approaches can be easily compared and measured. 
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2, Use of Subject Matter Experts (SMEs) 

It would be good to have two SMEs per IDS subgroup who will comment on and 
guide discussions during the simulation. Even more importantly, they can conduct after 
action review (AAR) sessions and speak from their experiences. The CCM TDS was 
developed with doctrinally sound infantry TTPs, but feedback from the system in this 
regard was not presented to the users during our experiment. Infantry SMEs are an 
essential part of training when using a TDS. 

3, Integrated Tactics and Planning Simulation Exercise (ITAPSE) 

We strongly suggest combining the best features of both the TDS and the TDG 
into TAPSE. This approach would incorporate the superior methodical planning 
techniques of the TDG with the ability to implement one’s plan and receive feedback 
provided by the TDS. Dr. Sadagic (2007) of the Naval Postgraduate School, MOVES 
Institute states, “...practitioners should be encouraged and offered time to explore a 
synergy of old (proven) training environments they know very well and simulations.” 
This quote, and our findings in Experiment 2, underscores the need to build a combined 
approach to tactical decision-making training. 

4, Focus on Planning 

We recommend that planning be examined in two different ways: between a TDG 
and TDS group and between a traditional TDS group and one that implemented ITAPSE. 

Another suggestion would be to vary the planning time for each group and, also, 
to look for correlations between which participants quickly gave up their primary plan 
compared to the time they spent creating their plan. Record the reasons participants 
provide for their decision to change their plans. 

Euture researchers might consider evaluating how quickly participants change 
their plans if they did not seem to be working. Then, record the rationale that led them to 
believe that their plans were not working. 

Einally, they might evaluate whether participants tended to stick with their 
original plan or if there was a tendency to disregard it at the earliest opportunity. Record 
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the time it took participants to realize that their original plan was not working and that the 
situation required a shift to an alternative COA. If they chose to deviate from their plan, 
did they have a justifiable reason to do so? 

5. Examine Personality Factors 

TDS participants had trouble keeping track of which units were friendly and 
which were enemy. Future studies could consider evaluating personality traits to 
determine whether there is a correlation with a participant’s tendency to rush into battle 
or to dig-in and wait. 

6. TDS After Action Review (AAR) 

Incorporate an after-action review session into each TDS session. A sufficient 
amount of time should be included in the sessions in order to accommodate this 
discussion. 

We believe that a thorough examination of the previous items, within the context 
of a follow-on study, could yield many interesting results such as a correlation between 
extroversion and perception of leadership performance and/or a successful outcome 
within the simulation. Additionally, future research might determine that a lengthy 
planning session correlates to more resistance to changing the plan upon enemy contact. 
Finally, the ITAPSE method of training decision-making skills could offer students the 
best that TDG and TDS to offer. 
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APPENDIX A - CONSENT FORM (EXPERIMENT 1) 


Naval Postgraduate School 

Participant Consent Form & 

Minimal Risk Statement 

Introduction. You are invited to partieipate in a study entitled Evaluation of the Preliminary Training for 
Virtual Simulations being condueted by the Naval Postgraduate Sehool MOVES Institute. 

Procedures. We will use the Human Systems Integration Laboratory (HISL) to eonduet an experiment 
examining three different training methods. We will evaluate training approaehes for operating a Taetieal 
Deeision-Making Simulation (TDS). The TDS is called Close Combat Marine (CCM) version 4.0 and is a 
virtual environment (a simulation) used by the USMC to train Marines in tactical decision-making. The 
researchers will provide the participants with a brief (no more than 5 minutes) explanation of the TDS and 
what they will be doing during the simulation. We will collect user computer simulation experience and 
demographics in the form of a questionnaire prior to the commencement of base I. 

Phase I will consist of a learning scenario where participants will learn how to operate the TDS. This phase 
will last approximately 30 minutes. 

In Phase II, the participants will go through a scenario a second time. 

In the final phase, Phase III, the researchers will administer a participant survey. Each participant will go 
through only one of the scenarios for Phase I and one scenario for phase II, a total of 2 scenarios. 

The researchers will record participants’ actions using a video camera (Note: participants themselves will 
not appear in the video). 

Risks and Benefits. I understand that this project does not involve greater than minimal risk and involves 
no known reasonably foreseeable risks or hazards greater than those encountered in everyday life. I have, 
also, been informed of any benefits to myself or to others that may reasonably be expected as a result of 
this research. 

Compensation. I understand that no tangible reward will be given. I understand that a copy of the 
research results will be available at the conclusion of the experiment. 

Confidentiality & Privacy Act. I understand that all records of this study will be kept confidential and 
that my privacy will be safeguarded. No information will be publicly accessible which could identify me 
as a participant and I will be identified only as a code number on all research forms. I understand that 
records of my participation will be maintained by NPS for 5 years, after which they will be destroyed. 

Voluntary Nature of the Study. I understand that my participation is strictly voluntary and, if I agree to 
participate, I am free to withdraw at any time without prejudice. 

Points of Contact. I understand that if I have any questions or comments regarding this project upon the 
completion of my participation, I should contact the Principal Investigator, C. Neil Fitzpatrick, 
Capt/USMC, (831) 917-6663, cnfitzpa@nps.edu , or Umit Ayvaz, Capt/Turkish Army, uayvaz@nps.edu. 
Any medical questions should be addressed to ETC Eric Morgan, MC, USA, (CO, POM Medical Clinic), 
(831) 242-7550, eric.morgan@nw.amedd.army.mil. 
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statement of Consent. I have read and understand the above information. I have asked all questions and 
have had my questions answered. I agree to partieipate in this study. I will be provided with a eopy of this 
form for my reeords. 

Partieipant’s Name (please print) 

Partieipant’s Signature Date 

Researeher’s Signature Date 
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APPENDIX B - IRB REQUEST (EXPERIMENT 1) 



C. Neil Fitzpatrick, Capt/USMC 
Umit Ayvaz, Capt/Turkish Army 
MOVES 
Watkins Hall 

Naval Postgraduate School 
Monterey, California 93943 


cnfitzpa@nps.edu 

(831) 917-6663 

uay vaz@nps .edu 

(831)-393-1272 


To; Protection of Human Subjects Committee 

Subject; Application for Human Subjects Review (Title); The most appropriate time and 
method for becoming comfortable with the Tactical Decision-Making Simulation (TDS) 
controls. 

1. Attached is a set of documents outlining a proposed experiment to be conducted 
by C. Neil Fitzpatrick (Captain USMC ) and Umit AYVAZ (Captain Turkish Army) 
for their final project in OA3402 Research Methods in Performance Assessment class 
in HSIL lab in Glasgow Hall. 

2. We are requesting approval of the described experimental protocol. For your 
reference, an experimental outline is included that describes the methods and 
measures we plan to use. 

3. We include the consent forms, privacy act statements, all materials and forms 
that a subject will read or fill-out, and the debriefing forms (if applicable) we will be 
using in the experiment. 

4. We understand that any modifications to the protocol or instruments/measures 
will require submission of updated IRB paperwork and possible re-review. Similarly, 
we understand that any untoward event or injury that involves a research participant 
will be reported immediately to the IRB Chair and NPS Dean of Research. 


Dr. Amela Sadagic, Principal Investigator (PI) 
C. Neil Fitzpatrick, Student Researcher 
limit AYVAZ, Student Researcher 
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APPLICATION FOR 

HUMAN SUBJECTS REVIEW (HSR) 


HSR NUMBER (to be assigned) 


PRINCIPAL INVESTIGATOR(S) (Full Name, Code, Telephone) 

Dr. Amela Sadagic, MOVES Research Associate Professor, 831.656.3819 


APPROVAL REQUESTED 

[ X ] New 

[ ] Renewal 

LEVEL OE RISK [ ] Exempt 
Justification: 

[ X ] Minimal [ ] More than Minimal 

WORK WILL BE DONE IN 
Human Systems Integration Lab. 

GLASCOW 

ESTIMATED NUMBER OE DAYS TO 
COMPLETE 

5 days 

MAXIMUM NUMBER OF SUBJECTS 

15 

ESTIMATED LENGTH OE EACH 
SUBJECT’S PARTICIPATION 

300 minutes 


SPECIAL POPULATIONS THAT WILL BE USED AS SUBJECTS 

[ ] Subordinates [ ] Minors [ X ] NPS Students [ ] Special Needs (e.g. Pregnant 
women) 


Specify safeguards to avoid undue influence and protect subject’s rights: 


OUTSIDE COOPERATING INVESTIGATORS AND AGENCIES 

[ ] A copy of the cooperating institution’s HSR decision is attached. _ 

TITLE OE EXPERIMENT AND DESCRIPTION OE RESEARCH (attach additional sheet 
if needed). 

Introduction to Tactical Decision-Making Simulations ITDS): Pointing and Clicking 

INTRODUCTION 

We will use the Human Systems Integration Laboratory (HISL) to conduct an experiment 
examining 3 different training methods. We will evaluate training approaches for operating a 
Tactical Decision-Making Simulation (TDS). The TDS is called Close Combat Marine 
(CCM) version 4.0 and is a virtual environment (a simulation) used by the USMC to train 
Marines in tactical decision-making. Researchers will provide the participants with a brief 
(no more than 5 minutes) explanation of the TDS and how they will participate in the 
simulation. 

We will administer and collect user computer simulation experience and demographics in the 
form of a Pre-TDS questionnaire prior to the commencement of Phase I. 

The overall study will be divided into 3 phases: 
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PHASE I 

Participants will engage in one of the 3 TDS training methodologies to learn how to operate 
the TDS. Phase I seenarios will inelude; 

a) TDS Boot Camp seenario alone (provided by the TDS manufaeturer), 

b) TDS Boot Camp seenario with memory-sheet, 

c) Memory-sheet alone with a praetiee scenario (user progresses at his own pace and 
asks questions of researehers if he gets stuek). 

This phase will last approximately 30 minutes. 

PHASE II 

All partieipants, regardless of Phase I grouping, will all go through the same phase II 
seenarios to provide a baseline for eomparison of the different groups. 

PHASE III 

Phase III is the final phase of the study. The researehers will administer a Post-TDS 
partieipant survey. Eaeh partieipant will go through only one of the scenarios for Phase I and 
one seenario for Phase II for a total of 2 seenarios. 

The researehers will reeord partieipants’ aetions using a video eamera (Note: partieipants 
themselves will not appear in the video.) The reeording of sessions will be both audio and 
video. Again, the user’s image will not appear in the video. The video will record the 
eomputer sereen that the partieipant/user is interaeting. This video will be used afterwards to 
evaluate the effieaey of the training method. The number of mistakes will be one of the 
objeetive me tries used to determine how well the participants learned the task. 

The researehers will, also, subjeetively evaluate eaeh partieipant on his/her performanee 
using a Eikert scale. 


I have read and understand NPS Notice on the Proteetion of Human Subjeets. If there are any 
ehanges in any of the above information or any ehanges to the attaehed Protoeol, Consent 
Eorm, or Debriefing Statement, I will suspend the experiment until I obtain new Committee 
approval. 

SIGNATURE_, PI 

SIGNATURE_, Student Researeher 

SIGNATURE_, Student Researeher 

DATE 
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APPENDIX C - SURVEY (EXPERIMENT 1) 


Pre-TDS Survey 

Please fill in the following questionnaire. Answer all questions as objectively as you can. All 
information will be held confidential. 

If you need to expand any answer please use the reverse side of your paper sheet. 


SIMULATION EXPERIENCE 


1. Were you required to use computer-based training simulations as part of your training in the past? 
YES / NO (circle one) 
a. If YES: 


1) What type simulations have you used and what skills were they used to train? 


1. Tactical level 

2. Operational level 

3. Strategic level 

Describe it: 

Describe it: 

Describe it: 


2) Tactical: What was the total number of hours you were expected to use them (estimate)? 
_hours 

Operational: What was the total number of hours you were expected to use them (estimate)? _ 

hours 

Strategic: What was the total number of hours you were expected to use them (estimate)? _hours 

3) That number of hours was (circle one number on the scale from 1 to 5) to acquire a good skill level: 


1 = not at all 

2 = less than 

3 = sufficient 

4 = more than 

5 = Totally 
sufficient 

sufficient 

sufficient 


sufficient 


4) Add any comment you’d like to justify that answer: 


2. Do you have experience with playing commercial video games? YES / NO (circle one) 

3. IfYES, 
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a) What kind of video games do you piay? (circie aii that appiy) 


1. first 

2. flight 

3. 

4. 

5. puzzle, 

6. 

7. online 

8. 

9. 

person 

simulations 

racing 

other 

strategy. 

adventure. 

multiplayer 

arcades 

other 

shooter 



sports 

card, board 

fantasy, role 
playing 

games 




b) How often do you piay them? 

Circie one: several times a week / once a week / several times a month / once a month / several times a 
year 

c) That is about_(number) hours per: day / week / month / year (circie one). 

4. Do you have your own personai computer or access to a computer that you can use on a daiiy basis? YES 
/ NO (circie one) 

a. If YES, how iong have you been using it on daiiy basis? For_(number) years and 

_(number) months. 

5. What hand do you use to operate a computer mouse? LEET / RIGHT / I am good with either (circle 
one) 


6. How often do you use email, browse the web, or use some other computer applications? (circle the answer in 
bold that applies to you) 



Knowledge and skills: 

Your current level 

Monthly Daily 

(Infrequent) (Very Frequent) 

A. 

Email 

after school hours 
(private use) 

1 

2 

3 

4 

5 


at school (in the unit) 

1 

2 

3 

4 

5 

B. 

Browse the web 

after school hours 
(private use) 

1 

2 

3 

4 

5 


at school (in the unit) 

1 

2 

3 

4 

5 

C. 

Office automation (Word, Excel, 
PPT) 

after school hours 
(private use) 

1 

2 

3 

4 

5 


at school (in the unit) 

1 

2 

3 

4 

5 

D. 

(some other application - which?) 

after school hours 
(private use) 

1 

2 

3 

4 

5 


at school (in the unit) 

1 

2 

3 

4 

5 

E. 

(some other application - which?) 

after school hours 
(private use) 

1 

2 

3 

4 

5 


at school (in the unit) 

1 

2 

3 

4 

5 
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7. When you think about your own knowledge of the procedures, and your current skills, how would you rank 
them? 


# 

Knowledge and skills: 

Your current level 

worst 

best 

1. 

Familiarity with a five paragraph order 

1 

2 

1 3 

4 

5 

2. 

Employ Machine guns in support of 
Offensive Operations 

1 

2 

1 3 

1 

4 

5 

3. 

Employ Machine guns in support of 
Defensive Operations 

1 

2 

1 3 

1 

4 

5 

4. 

Conduct a Infantry Squad/Platoon Attack 

1 

2 

1 3 

4 

5 

5. 

Control tactical movement of a 

Squad/Platoon in the Offense 

1 

2 

1 3 

1 

4 

5 

6. 

Conduct a Squad/Platoon Attack on Urban 
Terrain 

1 

2 

1 3 

1 

4 

5 

7. 

Conduct a Squad/Platoon Defense on Urban 
Terrain 

1 

2 

1 3 

1 

4 

5 


DEMOGRAPHICS 


1. Subject# = (You will be provided with a uniquely coded number):_ 

2. Date:_ Time:_ 

3. Year of birth: _ 

4. Country of Citizenship:_ 

5. Service component: (circle one) Army Navy Air Force Marine Corps Coast 

Guard Civilian 

6. Briefly describe your Primary MOS, (Example: Infantry, Surface Warfare, communications, etc): 


7. NPS Curriculum:_ 

8. Your Rank: _ 

9. Total number of months in service: _ 

Thank you very much for participating in the study. Relax and Enjoy! 
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Post-TDS Survey 


1. Date:_ Time:_ 

2. Subject# = (You will be provided with a uniquely coded number):_ 

3. What training methodology was administered during your session: (please check only one block) 

1. Group 1 

2. Group 2 

3. Group 3 

4. That amount of time for me to acquire a skill level adequate to participate in the simulation was: 
(Circle one number on the scale from 1 to 5.) 


1 = not at all 

2 = less than 

3 = sufficient 

4 = more than 

5 = Totally 

sufficient 

sufficient 


sufficient 

sufficient 


5. If you did not have enough time to become familiar with the TDS, how much more time do you believe you 
would need? 


6. How satisfied were you with your overall performance while using the CCM TDS: 


1 = not at all 

2 = less than 

3 = satisfied 

4 = somewhat 

5 = Extremely 
satisfied 

satisfied 

satisfied 


satisfied 


7. When you think about your own knowledge of tactical procedures and your current skill level, how would 
you rank them now after the TDS scenario? 


# 

Knowledge and skills: 

. Your current level 

worst best 


Familiarity with a five paragraph order 

1 

2 

3 

4 

5 


Employ Machine guns in support of Offensive 
Operations 

1 

2 

3 

4 

5 


Employ Machine guns in support of Defensive 
Operations 

1 

2 

3 

. .. . 

4 

5 


Conduct a Infantry Squad/Platoon Attack 

1 

2 

3 

4 

5 


Control tactical movement of a Squad/Platoon in 
the Offense 

1 

2 

3 

4 

5 


Conduct a Squad/Platoon Attack on Urban Terrain 

1 

2 

_ 

3 

4 

5 


Conduct a Squad/Platoon Defense on Urban Terrain 

1 


3 

4 

5 


Thanks very much for participating! 
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APPENDIX D - CCM MEMORY SHEET (EXPERIMENT 1) 


Close Combat Marines (CCM) 

Desired Action 


Selection/Deselect Units 

Select Troops Left Click on Unit 

Unselect Troops Left Click on another part of the screen 


Issue Orders Right click on Unit to displasrthe orders menu >> Left Click on desired order 

(Only orders that are available will be displayed.) Adot is placed on the battlefield to indicate the 
location/destination of your order. 


' MOVE 
i' MOVEFASt 
» MOVE COVERT 
F FIRE 
i' SMOKE 
DEFEND 
» AMBUSH 
• HOlWr/MSHOiJT' 
' DIGIN 


Move - Team members will walk 
Move Fast - Team members will run 

Move Covert -Team member will crawl and then defend when they arrive at destination 
Fire - Commands y ourteam to commence firing. Left clfck on target. 

Bright Green line: Your team can see Is target. 

Dark Green line: Sight blocked, but can still shoot. 

Red line: Shoot is blocked 

Smoke - Delivers smoke to a location to screen movements 
Defend - Troops set into a defensive posture 
Ambush - Troops find cover and fire only if enerny is very close 
Mount/Dismount -either get irrto or out of vehicles 
Dig - Troops dig and get into a fighting hole 


Screen Navigation 

Use ivbp Monitor (mumbnail Display) Left Click on the small map to jump to any location on the battlefield. 
Use Mouse and Keyboard 

Move Mewpoint Up Up Arow or move cursor to top edge of screen 
Move Mewpoirrt Down Down Anew or move cursor to bottom edge of screen 
Move Mewpoirrt Right Right Arow or move cursor to right edge of screen 
Move Mewpoirrt Left Left Arow or move cursor to left edge of screen 


Display Detailed Terrain Information - Right cibk entertain and HOLD. Message will appear in lower 
left comer. 

Toggle Trees on/off Ctrf-rt 


Monitors and Toolbars 

(Al game information is located at bottom of screen in toolbars. 

Turn them on S off with buttons at bottom left portion of the toolbar) 

You can: 

Monitor Morale 

Conduct Ar Strikes with Fixed Wlng/Rotary Wing 

(Left click on the Ar Icon.then left click on the location where it is to be delivered.) 
Conduct Atillery Barrage 

(Left Click Atillery Icon and then left click where you want ordinance delivered on screen 
Conduct Mortar Barrage 

Monitor Team Status (Left Click on team to get individual team member's: 

-names 

■health and morale 
-billet. 
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APPENDIX E - STATISTICAL ANALYSIS (EXPERIMENT 1) 


Experiment Hypothesis Test 

Hg = There is NO differenee between the methods. 

= There is a differenee between the methods. 

We used F (ANOVA) test for our first experiment. This is beeause we had 3 
groups of people. The researehers ehose an a = 0.05 as the signifieanee eriteria whether 
to rejeet the null hypothesis. 

Since p = 0,2241, which is greater than a = 0,05, we are not able to reject Ho, We 
conclude that there is no significant difference between the 3 methods. 


Methods vs. Total Errors 



Analysis of Variance 


Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

2 

136.50000 

68.2500 

1.1601 

Error 

9 

529.50000 

58.8333 

Prob > F 

C. Total 

11 

666.00000 


0.3563 
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Method vs. Total Error Factor 


Whole Model 

Actual by Predicted Plot 
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Total Error Factor Predicted 
P=0.2241 RSq=0.28 RMSE=5.4508 


Analysis of Variance 


Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

2 

105.42002 

52.7100 

1.7741 

Error 

9 

267.39828 

29.7109 

Prob > F 

C. Total 

11 

372.81829 


0.2241 
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Other Findin2s 


For the following plots, we found that there was a signifieant eorrelation between Gender 
and Total Error Faetor, as well as, Gaming Experienee and Total Error Faetor. 

Gender vs. Total Error Factor 



Analysis of Variance 


Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

1 

119.06447 

119.064 

4.6921 

Error 

10 

253.75382 

25.375 

Prob > F 

C. Total 

11 

372.81829 


0.0555 


Gaming Experience vs. Total Error Factor 
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Analysis of Variance 


Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

1 

177.42417 

177.424 

9.0803 

Error 

10 

195.39412 

19.539 

Prob > F 

C. Total 

11 

372.81829 


0.0130 


Simulation Experience 


Actual by Predicted Plot 
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Total Error Factor Predicted 
P=0.2913 RSq=0.11 RMSE=5.759 


Analysis of Variance 


Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

1 

41.15848 

41.1585 

1.2410 

Error 

10 

331.65981 

33.1660 

Prob > F 

C. Total 

11 

372.81829 


0.2913 


Time In Service (TIS) 


Actual by Predicted Plot 
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Total Error Factor Predicted 
P=0.7141 RSq=0.01 RMSE=6.063 
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Analysis of Variance 


Source 

DF 

Sum of Squares 

Mean Square 

F Ratio 

Model 

1 

5.22407 

5.2241 

0.1421 

Error 

10 

367.59423 

36.7594 

Prob > F 

C. Total 

11 

372.81829 


0.7141 
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APPENDIX F - CONSENT FORM (EXPERIMENT 2) 


Naval Postgraduate School 

Participant Consent Form & 

Minimal Risk Statement 

Introduction. You are invited to participate in a study that will compare the training effectiveness 
of a Tactical Decision-Making Simulation (TDS) and a Tactical Decision-Making Game (TDG). 
This will be conducted by the Naval Postgraduate School Modeling of Virtual Environments and 
Simulation (MOVES) Institute. 

Procedures. We will use a lab in Watkin s Hall (212a) to conduct this experiment examining 
these 2 different training approaches. The TDS that will be used is called Close Combat Marine 
(CCM) version 4.0 and is a virtual environment (a simulation) used by the USMC to train 
Marines in tactical decision-making. The researchers will provide the participants with a briefing 
during Session I. This will describe the number of sessions and the basic skills that will be 
necessary to complete both the TDS and the TDG. We will collect user computer simulation 
experience and demographics in the form of a questionnaire prior to Session I. Participants will 
be divided into a TDG group (8 participants) and a TDS group (8 participants) and remain in 
those groups for the duration of the experiment. Each of these will be further subdivided into 2 
groups of 4 each. 

Session I TDS and TDG will be centered on a tactical situation within a small town. Participants 
will be taught the necessary tactics that will be used in the sessions. In the TDS session, 
participants will be informed about what the TDS is and how to operate it. The TDG group will be 
introduced to the nature of a TDG by participating in an abbreviated TDG session. Both TDS and 
TDG sessions will last approximately 60 minutes for both groups. 

In Sessions 2, 3, 4, and 5, the TDS participants will go through a different scenario each time. TDS 
users will alternate between playing the offense and the defense, changing roles after each session. 
TDG users will all be offense or defense during a single session and switch roles for the next 
session. The researchers will administer questionnaires after each session. 

The researchers will record some of the participants’ actions using a video camera, but will not tie 
user demographic information to the video. Participants will not be identifiable by name, country, 
etc. The video tape will be destroyed upon completion of the experiment. 

Risks and Benefits. I understand that this project does not involve greater than minimal risk and 
involves no known reasonably foreseeable risks or hazards greater than those encountered in 
everyday life. I have, also, been informed of any benefits to myself or to others that may 
reasonably be expected as a result of this research. 

Compensation. I understand that participation in this study will earn me some class credit in my 
OA3101 course and that the amount of my participation will be provided to my instructor (Col E. 
Shattuck). A copy of the research results will be available at the conclusion of the experiment if I 
would like to have a copy of them. 
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Confidentiality & Privacy Act. I understand that all records of this study will be kept confidential 
and that my privacy will be safeguarded. No information will be publicly accessible which could 
identify me as a participant, and I will be identified only as a code number on all research forms. I 
understand that records of my participation will be maintained by NPS for 5 years, after which they 
will be destroyed. 

Voluntary Nature of the Study. I understand that my participation is strictly voluntary, and if I 
agree to participate, I am free to withdraw at any time without prejudice. 

Points of Contact. I understand that if I have any questions or comments regarding this project 
upon the completion of my participation, I should contact the Principal Investigator, C. Neil 
Fitzpatrick, Capt/USMC, (831) 917-6663, cnFitzpa@nps.edu, or limit Ayvaz, Capt/Turkish Army, 
uayvaz@nps.edu. Any medical questions should be addressed to LTC Eric Morgan, MC, USA, 
(CO, POM Medical Clinic), (831) 242-7550, eric.morgan@nw.amedd.army.mil. 

Statement of Consent. I have read and understand the above information. I have asked all 
questions and have had my questions answered. I agree to participate in this study. I will be 
provided with a copy of this form for my records. 


Participant’s Name (please print) 


Participant’s Signature 


Date 


Researcher’s Signature 


Date 
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APPENDIX G - IRB REQUEST (EXPERIMENT 2) 



Amela Sadagic, PhD 
C. Neil Fitzpatrick, Capt/USMC 
Umit Ayvaz, Capt/Turkish Army 
MOVES 
Watkins Hall 

Naval Postgraduate School 
Monterey, California 93943 


asadagic@nps,edu 
(831) 656-3819 

cnfitzpa@nps.edu 

(831)917-6663 

uayvaz@nps.edu 

(831)-393-1272 


To: Protection of Human Subjects Committee 

Subject: Comparison of the Training Effectiveness of a Tactical Decision-Making Simulation (TDS) and a 
Tactical Decision-Making Game (TDG). 

Attached is a set of documents outlining a proposed experiment to be conducted by C. Neil Fitzpatrick 
(Captain USMC ) and Umit AYVAZ (Captain Turkish Army) for their pilot study for their thesis in a Lab 
in Watkins Hall (212a). 

1. We are requesting approval of the described experimental protocol. An experimental outline is 
included for your reference that describes the methods and measures we plan to use. 

2. We include the consent forms, privacy act statements, all materials and forms that a subject will read 
or fill-out, and the debriefing forms (if applicable) we will be using in the experiment. 

3. We understand that any modifications to the protocol or instruments/measures will require submission 
of updated IRB paperwork and possible re-review. Similarly, we understand that any untoward event 
or injury that involves a research participant will be reported immediately to the IRB Chair and NFS 
Dean of Research. 

The work on this study is a part of NMSO sponsored project (R9D8S) with title: “Studying the Synergy of 
Virtual Training Simulations and Novel Training Methodologies for Training of the Ground Officers, and 
Evaluation of Training Effectiveness.” 


Dr. Amela Sadagic, Principal Investigator (PI) 


C. Neil Fitzpatrick, Student Researcher 


Umit AYVAZ, Student Researcher 
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APPLICATION FOR 

HUMAN SUBJECTS REVIEW (HSR) 


HSR NUMBER (to be assigned) 


PRINCIPAL INVESTIGATOR(S) (Full Name, Code, Telephone) 


Dr. Amela Sadagic, MOVES Research Associate Professor, 831.656.3819 


APPROVAL REQUESTED [ X ] New 

[ ] Renewal 

LEVEL OE RISK [ ] Exempt [ X ] Minimal [ ] More than Minimal 

Justification: The subjects in the study will be asked to use virtual training simulation on 
desktop computer (they will also use computer mouse and keyboard) and fill in several 
questionnaire forms. 

WORK WILL BE DONE IN Watkins Hall 
Lab in Watkins Hall (212a) 

ESTIMATED NUMBER OE DAYS TO 
COMPLETE 

30 days 

MAXIMUM NUMBER OE SUBJECTS 

16 

ESTIMATED LENGTH OE EACH 
SUBJECT’S PARTICIPATION 

300 minutes total (5 sessions where 1 
session is held on one day, 60 min each 
session) 


SPECIAL POPULATIONS THAT WILL BE USED AS SUBJECTS 

[ ] Subordinates [ ] Minors [ X ] NPS Students [ ] Special Needs (e.g. Pregnant 
women) 


Specify safeguards to avoid undue influence and protect subject’s rights: 

The participation in the study is voluntary and will not be imposed on any NPS student: each 
person will have an option of withdrawing from the study at any point in time. The data 
collected in the study will be safeguarded as requested by generally accepted IRB standards: 
each person will be identified only as a code number on all research forms/data bases; name 
of any person on any signed document will not be paired with their code number in order to 
protect their identity; and records of subject’s participation will be maintained by NPS for 3 
years, after which they will be destroyed. 


OUTSIDE COOPERATING INVESTIGATORS AND AGENCIES 
[ ] A copy of the cooperating institution’s HSR decision is attached. 
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TITLE OF EXPERIMENT AND DESCRIPTION OF RESEARCH (attach additional sheet 
if needed). 

COMPARISON OF THE TRAINING EFFECTIVENESS OF A TACTICAE DECISION¬ 
MAKING SIMUEATION (TPS) AND A TACTICAE DECISION-MAKING GAME 
(TDG) 

INTRODUCTION 

We will use the Eab in Watkins Hall (212a) to eonduet a study that will eompare the training 
effeetiveness of a Tactical Decision-Making Simulation (TDS) and a Tactical Decision- 
Making Game (TDG). The TDS is called Close Combat Marine (CCM) version 4.0 and is a 
virtual environment (a simulation) used by the USMC to train Marines in tactical decision 
making. 

Participants will be divided into a TDG group (8 participants) and a TDS group (8 
participants) and remain in those groups for the duration of the experiment. Each of these 
will be further subdivided into 2 groups of 4 eaeh. 

The overall study will be divided up into 5 sessions both for the TDG and the TDS group: 
SESSION 1 

The researchers will collect user computer simulation experience and demographics in the 
form of a questionnaire prior to Session I. The researchers will provide both TDG and TDS 
partieipants with a briefing during Session I. This will deseribe the number of sessions and 
the basie skills that will be neeessary to eomplete both the TDS and the TDG. In this 
session, TDS and TDG will be centered on a tactical situation within a small town. In the 
TDS session, participants will be informed about what the TDS is and how to operate it. The 
TDG group will be introdueed to the nature of a TDG by participating in an abbreviated 
TDG session. 

Both TDS and TDG sessions will last approximately 60 minutes. 

SESSION2. 3.4. AND5: 

The TDS participants will go through a different scenario each time. TDS users will 
alternate between playing the offense and the defense, changing roles after each session. 
TDG users will all be offense or defense during a single session and switeh roles for the next 
session. The researehers will administer questionnaires after each session, as well as a final 
questionnaire after Session 5 in both TDS and TDG. Both TDS and TDG sessions will last 
approximately 60 minutes. 

The researchers will reeord some of the participants’ actions using a video eamera, but will 
not tie user demographic information to the video. Participants will not be identifiable by 
name, country, etc. The video tape will be destroyed upon completion of the experiment, but 
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digital recordings (values recorded by the CCM training simulation) will be retained for the 
purposed of training and follow-on work. 

The researchers will, also, evaluate each participant on their ability to lead a group, to make 
tactically sound and timely decisions, and their situational awareness in the TDS and TDG 
scenarios. 


I have read and understand NFS Notice on the Protection of Human Subjects. If there are any 
changes in any of the above information or any changes to the attached Protocol, Consent 
Form, or Debriefing Statement, I will suspend the experiment until I obtain new Committee 
approval. 

SIGNATURE_, PI 

SIGNATURE_, Student Researcher 

SIGNATURE_, Student Researcher 

DATE 
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APPENDIX H - TDG SURVEY (EXPERIMENT 2) 


Pilot Study Survey 
- TDG Session 1 - 
Pre-questionnaire 

Please fill in the following questionnaire. Answer all questions as objectively as you can. All 

information will be held confidential. 

If you need to expand any answer please use the reverse side of your paper sheet. 

1) Were you required to use computer-based simulations as part of your training in the past? NO / YES 
(circle one) 

a) NO. 

b) If YES: What type simulations have you used and what skills were they used to train? 
i) Tactical Level (Marksmanship trainer, ship driving simulator, Language Trainer, etc.) 

(1) Description: 


(2) Estimate total number of hours you used it:_ 

(3) To gain competency, this amount of time spent was: (Circle one) 



i) Operational Level (OneSAF, JSAF, JCATS, Common Operational Picture of the Battlespace, etc.) 

(1) Description: 


(2) Estimate total number of hours you used it:_ 

(3) To gain competency, this amount of time spent was: (Circle one) 
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2) Do you have experience playing commercial video games? NO / YES (circle one) 

a) NO. 

b) If YES, What kind of video games do you play? (answer all that apply to you; circle day, week. 


month or year in each line) 

i) First person shooter. (Stand Alone or Online) 

year 

ii) Flight Simulation. (Stand Alone or Online) 
year 

iii) Racing. (Stand Alone or Online) 

year 

iv) Sports (Football, etc.) (Stand Alone or Online) 

year 

v) Puzzle, Card, Board (Stand Alone or Online) 

year 

vi) Strategy (Stand Alone or Online) 

year 

vii) Adventure, Fantasy (Stand Alone or Online) 

year 

viii) Arcade (Stand Alone or Online) 

year 

ix) Other (Stand Alone or Online) 

year 


Approximately 

hours ner day / week / month / 

Approximately 

hours per day / week / month / 

Approximately 

hours per day / week / month / 

Approximately 

hours per day / week / month / 

Approximately 

hours per day / week / month / 

Approximately 

hours per day / week / month / 

Approximately 

hours per day / week / month / 

Approximately 

hours per day / week / month / 

Approximately 

hours per day / week / month / 


3) Do you own a personal computer? NO / YES (circle one) 

a) NO. 

b) If YES, how often do you use it? Approximately_hours per day / week / month / year 

4) What hand do you use to operate a computer mouse? LEET / RIGHT / I am good with either (circle 
one) 

5) How often do you use the following other computer applications? (please answer all that apply to you; circle 
day, week, month or year in each line) 


i) E-mail: 

ii) Browse Web: 

iii) IMovie: 

iv) ITunes: 

v) Excel: 

vi) Word: 

vii) PowerPoint: 


Approximately 

Approximately 

Approximately 

Approximately 

Approximately 

Approximately 

Approximately 


hours per day / week / month / year 
hours per day / week / month / year 
hours per day / week / month / year 
hours per day / week / month / year 
hours per day / week / month / year 
hours per day / week / month / year 
hours per day / week / month / year 


6) When you think about your own knowledge of infantry tactics, techniques, and procedures, how would you 
rank them? (please circle NA or one number between 1 and 7 in each line) 
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# 

Knowledge and skills: 

Your current level 

Poor 

Excellent 

1. 

Familiarity with a five paragraph order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Employ Machine guns in support of Offensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Employ Machine guns in support of Defensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Infantry Offensive Operations in Field 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Infantry Defensive Operations in Field 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Tactical Movement On Objective 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Fire and Maneuver 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

Infantry Attack in Urban Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Infantry Defense in Urban Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Issue an Order 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Conduct Tactical Movement in the Offense 

NA 

1 

2 

3 

4 

5 

6 

7 


7) Subject# (your code name):_ 

8) Date:_ Time: 

9) Year of birth: _ 

10) Country of Citizenship:_ 

11) Service component: (Circle one) 


Army 

Navy 

Air Force 

Marine Corps 

Coast Guard 

Civilian 


12) Briefly describe your Primary MOS, (Example: Infantry, Surface Warfare, communications, etc.): 


13) NPS Curriculum:_ 

14) Your Rank: _ 

15) Time in Service:_years_months 

Thank you very much for participating in the study. 
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Pilot Study Survey 


- TDG Session 2 - 
Questionnaire 


Please fill in the following questionnaire. Answer all questions as objectively as you can. 
All information will be held in confidence. 

If you need to expand any answer please use the reverse side of your paper. 


Participant # = (Your code name):_ 

Date:_ Time:_ 

1) What training group were you in? (Circle One) 

a. Offense 

b. Defense 

2) For each person (including yourself) give a score between 0 and 100 which estimates the degree to which that 
person was the ‘leader’ in the last training session. (The four scores must add to 100). 

( For example , if no clear leader emerged, then each should have a score of 25. If Person X played the major role 
in this regard, then Person X would have a score of say 80, and the other 2 people, a score of 10 each. A score 
of 0 means that the corresponding person was in no way a leader; whereas a score of 100 means that the 
corresponding person was the clear and only leader.) 


Person (code name): 

Score/100: 









Total Score: 

100 


3) Think about a previous time when you enjoyed working together in a group. To what extent have you enjoyed 
the group experience just now? 

I enjoyed the group experience just now (please check one number between 1 and 7): 


1. Not at all 


It was nothing like the previous experience of working in a group. 

2. 



3. 



4. 



5. 



6. 



7. Very much so. 


I enjoyed it just as much as the previous experience of working in a group. 


4) Think back now about the session that you have just completed. For example, to what extent in your 
imagination can you move around that terrain (battlefield) now? 


I can think myself back on that terrain/battlefield (please check one number between 1 and 7): 
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1. Not at all 


I cannot imagine now at all being on that terrain/battlefield now. 

2. 



3. 



4. 



5. 



6. 



7. Very much so. 


I can strongly imagine being back on that terrain/battlefield now. 


5) In the last session, to what extent did you have the sense of the other three people being together with you ? 


In the last session, I had a sense that the other three people were with me (please check one number between 1 
and 7): 


1. Not at all 


I did not have a sense that the other three people were with me. 

2. 



3. 



4. 



5. 



6. 



7. Very much so. 


The other three people were very much with me. 


6) Who did most of the talking? Give a score to each person between 0 and 100, where the four scores add to 
100. (A person would be given a score of near 100 only if they did almost all the talking. They would be given a 
score of near 0 if they did almost no talking). 

The extent to which each person did most of the talking was: 


Person (code name): 

Score/100: 









Total Score: 

100 


7) Overall, how cooperative was each of the other three people (please check one value between 1 and 7 for 
each person). 


1 Person (code name) 





1. Not at all 




S/he was not cooperative at all 

2. 





3. 





4. 





5. 





6. 





7. Very much so. 




S/he was very cooperative 


8) Think back now about past session, what were good tactical decision and procedures done by your team? 
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9) Think back now about past session, what were bad tactical decision and procedures done by your team? 


10) Think back now about past session, what were good tactical decision and procedures done by other team? 


11) Think back now about past session, what were bad tactical decision and procedures done by other team? 
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12) What can you say about: 


a) Team Member #1, (Participant’s code name:_) where Poor (1) - Excellent (7). 

(please circle NA or one number between 1 and 7 in each line) 


# 

Knowledge and skills: 

. Your current level 

poor 

Excellent 

1. 

Understood Operations Order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Identified Enemy Most Probably Course of Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created a Scheme of Maneuver (SOM) 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Considers Adjacent Units 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Created a Fire Support Plan (FSP) 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Identified Secondary MPCOA 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Considered the following in Mission Planning 
Process: 









7.a 

- Mission 

NA 

1 

2 

3 

4 

5 

6 

7 

7.b 

- Enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

7.C 

- Troops Available 

NA 

L" 

2 

3 

4 

5 

6 

7 

7.d 

- Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

7.e 

- Time 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

Identified Enemy of Center of Gravity and 

Attacks It. 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Adjusted to Enemy Situation 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Took Charge 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Maintained Oversight of All Subordinate Units 

NA 

1 

2 

3 

4 

5 

6 

7 

12. 

Communicated Flis Plan/Intent to Subordinates 

NA 

1 

2 

3 

4 

5 

6 

7 

13. 

Ensured that Subordinates what they had been 
told 

NA 

1 

2 

3 

4 

5 

6 

7 
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# 

Knowledge and skills: 

. Your current level 

poor 

Excellent 

14. 

Used deception and creativity to outwit the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

15. 

Deviated from doctrine when necessary 

NA 

1 

2 

3 

4 

5 

6 

7 

16. 

Enemy Location 

NA 

1 

2 

3 

4 

5 

6 

7 

17. 

Size of Enemy Element 

NA 

1 

2 

3 

4 

5 

6 

7 

18. 

Direction of Enemy Movement 

NA 

1 

2 

3 

4 

5 

6 

7 

19. 

Understanding of Battle Space Geometry 

NA 

1 

2 

3 

4 

5 

6 

7 


b) Team Member #2, (Participant’s code name:_) where Poor (1) - Excellent (7). 

(please circle NA or one number between 1 and 7 in each line) 


# 

Knowledge and skills: 

, Your current level 

poor 

Excellent 

1. 

Understood Operations Order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Identified Enemy Most Probably Course of Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created a Scheme of Maneuver (SOM) 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Considers Adjacent Units 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Created a Fire Support Plan (FSP) 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Identified Secondary MPCOA 

NA 


2 

3 

4 

5 

6 

7 

7. 

Considered the following in Mission Planning 
Process: 









7.a 

- Mission 

NA 

1 

2 

3 

4 

5 

6 

7 

7.b 

- Enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

7.C 

- Troops Available 

NA 

1 

2 

3 

4 

5 

6 

7 

7.d 

- Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

7.e 

- Time 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

Identified Enemy of Center of Gravity and 

Attacks It. 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Adjusted to Enemy Situation 

NA 

1 

2 

3 

4 

L" 

6 

7 

10. 

Took Charge 

NA 


2 

3 

4 

5 

6 

7 

11. 

Maintained Oversight of All Subordinate Units 

NA 

1 

2 

3 

4 

5 

6 

7 

12. 

Communicated Flis Plan/Intent to Subordinates 

NA 

1 

2 

3 

4 

5 

6 

7 

13. 

Ensured that Subordinates what they had been 
told 

NA 

1 

2 

3 

4 

5 

6 

7 

14. 

Used deception and creativity to outwit the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 
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# 

Knowledge and skills: 

. Your current level 

poor 

Excellent 

15. 

Deviated from doctrine when necessary 

NA 

1 

2 

3 

4 1 

5 

1 6 

7 

16. 

Enemy Location 

NA 

1 

2 

3 

4 1 

5 

1 6 

7 

17. 

Size of Enemy Element 

NA 

1 

2 

3 

4 1 

5 

1 6 

7 

18. 

Direction of Enemy Movement 

NA 

1 

2 

3 

4 1 

5 

1 6 

7 

19. 

Understanding of Battle Space Geometry 

NA 

1 

2 

3 

4 1 

5 

1 6 

7 


c) Team Member #3, (Participant’s code name:_), where Poor (1) - Excellent (7). 

(please circle NA or one number between 1 and 7 in each line) 


# 

Knowledge and skills: 

, Your current level 

poor 

Excellent 

1. 

Understood Operations Order 

NA 

1 1 2 1 3 

4 

5 

6 

7 

2. 

Identified Enemy Most Probably Course of Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created a Scheme of Maneuver (SOM) 

NA 

112 13 

4 

5 

6 

7 

4. 

Considers Adjacent Units 

NA 

1 1 2 1 3 

4 

5 

6 

7 

5. 

Created a Fire Support Plan (FSP) 

NA 

1 1 2 1 3 

4 

5 

6 

7 

6. 

7. 

Identified Secondary MPCOA 

NA 

1 i 2 1 3 

4 

5 

6 

7 

Considered the following in Mission Planning 
Process: 









7.a 

- Mission 

NA 

112 13 

4 

5 

6 

7 

7.b 

- Enemy 

NA 

1 1 2 1 3 

4 

5 

6 

7 

7.C 

- Troops Available 

NA 

1 1 2 1 3 

4 

5 

6 

7 

7.d 

- Terrain 

NA 

112 13 

4 

5 

6 

7 

7.e 

- Time 

NA 

112 13 

4 

5 

6 

7 

8. 

Identified Enemy of Center of Gravity and 

Attacks It. 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Adjusted to Enemy Situation 

NA 

1 1 2 1 3 

4 

5 

6 

7 

10. 

Took Charge 

NA 

1 1 2 1 3 

4 

5 

6 

7 

11. 

Maintained Oversight of All Subordinate Units 

NA 

1 1 2 1 3 

4 

5 

6 

7 

12. 

Communicated Flis Plan/Intent to Subordinates 

NA 

112 13 

4 

5 

6 

7 

13. 

Ensured that Subordinates what they had been 
told 

NA 

1 

2 

3 

4 

5 

6 

7 

14. 

Used deception and creativity to outwit the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

15. 

Deviated from doctrine when necessary 

NA 

1 1 2 1 3 

4 

5 

6 

7 
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# 

Knowledge and skills: 

. Your current level 

poor 

Excellent 

16. 

Enemy Location 

NA 

1 

2 

3 

4 

5 

6 

7 

17. 

Size of Enemy Element 

NA 

1 

2 

3 

4 

5 

6 

7 

18. 

Direction of Enemy Movement 

NA 

1 

2 

3 

4 

5 

6 

7 

19. 

Understanding of Battle Space Geometry 

NA 

1 

2 

3 

4 

5 

6 

7 


13) YOURSELF on the following criteria where Poor (1) - Excellent (7). (please circle NA or one number 
between 1 and 7 in each line) 


# 

Knowledge and skills: 

, Your current level 

poor 

Excellent 

1. 

Understood Operations Order 

NA 

1 1 2 1 3 

4 

5 

6 

7 

2. 

Identified Enemy Most Probably Course of Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created a Scheme of Maneuver (SOM) 

NA 

1 1 2 1 3 

4 

5 

6 

7 

4. 

Considers Adjacent Units 

NA 

1 1 2 1 3 

4 

5 

6 

7 

5. 

Created a Fire Support Plan (FSP) 

NA 

1 1 2 1 3 

V 

5 

6 

7 

6. 

Identified Secondary MPCOA 

NA 

1 1 2 1 3 

4 

5 

6 

7 

7. 

Considered the following in Mission Planning 
Process: 









7.a 

- Mission 

NA 

1 1 2 1 3 

4 

5 

6 

7 

7.b 

- Enemy 

NA 

1 i 2 1 3 

4 

5 

6 

7 

7.C 

- Troops Available 

NA 

1 1 2 1 3 

4 

5 

6 

7 

7.d 

- Terrain 

NA 

112 13 

4 

5 

6 

7 

7.e 

- Time 

NA 

112 13 

4 

5 

6 

7 

8. 

Identified Enemy of Center of Gravity and 

Attacks It. 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Adjusted to Enemy Situation 

NA 

1 1 2 1 3 

4 

5 

6 

7 

10. 

Took Charge 

NA 

1 1 2 1 3 

4 

5 

6 

7 

11. 

Maintained Oversight of All Subordinate Units 

NA 

1 1 2 1 3 

4 

5 

6 

7 

12. 

Communicated Flis Plan/Intent to Subordinates 

NA 

112 13 

4 

5 

6 

7 

13. 

Ensured that Subordinates what they had been 
told 

NA 

1 

2 

3 

4 

5 

6 

7 

14. 

Used deception and creativity to outwit the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

15. 

Deviated from doctrine when necessary 

NA 

1 1 2 1 3 

4 

5 

6 

7 

16. 

Knew Enemy Location 

NA 

1 1 2 1 3 

4 

5 

6 

7 
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# 

Knowledge and skills: 

. Your current level 

poor 

Excellent 

17. 

Knew the Size of the Enemy Element 

NA 

1 

2 

1 3 

4 

5 

6 

7 

18. 

Knew the Direction of the Enemy's Movement 

NA 

1 

2 

1 3 

4 

5 

6 

7 

19. 

Understood the Battle Space Geometry 

NA 

1 

2 

1 3 

4 

5 

6 

7 


14) If you did not have enough time to become familiar with the infantry tactics of the offense or defense 
(whichever you participated in), how much more time do you believe you would need? 


15) How satisfied were you with your overall performance while participating in the TDG: 


1 =Not 

2 

3 

4 =Average 

5 

6 

7 = 

Satisfied 





Extremely 


Thanks very much for participating! 

NOTE: Questionnaires for Sessions 3-5 are identical to Session 2 (above). 
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Pilot Study Survey 


- TDG FINAL - 
Questionnaire 

Please fill in the following questionnaire. Answer all questions as objectively as you can. 

All information will be held in confidence. 

If you need to expand any answer please use the reverse side of your paper sheet. 

1) Which of the following items/drills did you exercise at least once during the TDG sessions? (Circle all that 
apply from the list below.) 


# 

Knowledge and skills: 

1. 

Familiarity with a five paragraph order 

2. 

Employ Machine guns in support of Offensive Operations 

3. 

Employ Machine guns in support of Defensive Operations 

4. 

Infantry Offensive Operations in Field 

5. 

Infantry Defensive Operations in Field 

6. 

Tactical Movement On Objective 

7. 

Fire and Maneuver 

8. 

Infantry Attack in Urban Terrain 

9. 

Infantry Defense in Urban Terrain 

10. 

Issue an Order 

11. 

Conduct Tactical Movement in the Offense 


2) Of the actions that you chose in the previous question (Question 1), with which did you have the most 
difficulty ? 

(circle three) 


# 

Knowledge and skills: 

1. 

Familiarity with a five paragraph order 

2. 

Employ Machine guns in support of Offensive Operations 

3. 

Employ Machine guns in support of Defensive Operations 

4. 

Infantry Offensive Operations in Field 

5. 

Infantry Defensive Operations in Field 

6. 

Tactical Movement On Objective 
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# 

Knowledge and skills: 

7. 

Fire and Maneuver 1 

8. 

Infantry Attack in Urban Terrain 1 

9. 

Infantry Defense in Urban Terrain 1 

10. 

Issue an Order i 

11. 

Conduct Tactical Movement in the Offense 1 

_1 


3) Of the actions that you chose in the Question 1, which do you feel you improved upon most during the TDG 
sessions? (circle three) 


# 

Knowledge and skills: 

1. 

Familiarity with a five paragraph order 

2. 

Employ Machine guns in support of Offensive Operations 

3. 

Employ Machine guns in support of Defensive Operations 

4. 

Infantry Offensive Operations in Field 

5. 

Infantry Defensive Operations in Field 

6. 

Tactical Movement On Objective 

7. 

Fire and Maneuver 

8. 

Infantry Attack in Urban Terrain 

9. 

Infantry Defense in Urban Terrain 

10. 

Issue an Order 

11. 

Conduct Tactical Movement in the Offense 


4) When you think about your own knowledge of infantry tactics, techniques, and procedures, how would you rank 
them? (please circle NA or one number between 1 and 7 in each line) 


# 

Knowledge and skills: 

Your current 

level 

Poor 

Excellent 

1. 

Familiarity with a five paragraph order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Employ Machine guns in support of Offensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Employ Machine guns in support of Defensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Infantry Offensive Operations in Field 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Infantry Defensive Operations in Field 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Tactical Movement On Objective 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Fire and Maneuver 

NA 

1 

2 

3 

4 

5 

6 

7 
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# 

Knowledge and skills: 


level 



Your current 




Poor 

Excellent 






8. 

Infantry Attack in Urban Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Infantry Defense in Urban Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Issue an Order 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Conduct Tactical Movement in the Offense 

NA 

1 

2 

3 

4 

5 

6 

7 


5) When you think about your own knowledge of infantry tactics, techniques, and procedures, how much did 
you improve in each? (please circle NA or one number between 1 and 7 in each line) 


# 

Knowledge and skills: 

Your improvement 

They did 

Improved 

Improve at 

greatly 

not 

all 

1. 

Familiarity with a five paragraph order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Employ Machine guns in support of Offensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Employ Machine guns in support of Defensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Infantry Offensive Operations in Field 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Infantry Defensive Operations in Field 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Tactical Movement On Objective 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Fire and Maneuver 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

Infantry Attack in Urban Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Infantry Defense in Urban Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Issue an Order 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Conduct Tactical Movement in the Offense 

NA 

1 

2 

3 

4 

5 

6 

7 


4) Rate the TDG experience on its ability to provide you with a good mental picture of the battle space: 


1 = poor 

2 

3 

4 = average 

5 

6 

1 7 = excellent 


5) Rate the following statements (circle one number for each statement): 


“During the TDG sessions, I felt like I was 
taking part in training.” 


1 = 

strongly 

disagree 

2 

3 

4 = 

neutral 

5 

6 

7 = 

strongly 

agree 

“During the TDG sessions, I felt like my 
actions/my plan had no consequences to 
the plans of the group.” 


1 = 

strongly 

disagree 

2 

3 

4 = 

neutral 

~5 

6 

7 = 

strongly 

agree 

“During the TDG sessions, I felt like I was 
part of the group working together.” 


1 = 

strongly 

disagree 

2 

3 

4 = 

neutral 

5 

6 

7 = 

strongly 

agree 
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4. 

“During the TDG sessions, I felt like I was 
playing a game.” 


1 = 

strongly 

disagree 

2 

3 

4 = 

neutral 

5 

6 

7 = 

strongly 

agree 

5. 

“During the TDG sessions, I felt isolated 
from the others.” 


1 = 

strongly 

disagree 

2 

3 

4 = 

neutral 

5 

6 

7 = 

strongly 

agree 


6) In your opinion what was the overall quality of training in the TDG? (circle one) 


L_ 

1 = not satisfactory 

2 

3 4 = average 5 

6 

7 = excellent 

7) 

Throughout all the TDG 

sessions, do you believe that you made good tactical decisions? (circle one number) 


1 = poor 

2 

3 1 4 = satisfactorily 5 

6 

7 = excellent 

8) How well did the TDG sessions meet your expectations regarding training value? 

L 

1 = not satisfactorily 

2 

3 14= satisfactorily 1 5 

6 

7 = exceptionally 
well 


9) When you think about your future training needs, to what extent would you like it to be supplemented with 
TDGs? 


j 1 = not satisfactorily 

2 

3 

1 4 = satisfactorily 

5 

6 

7 = exceptionally 

1 






well 


10) When you think about your future training needs, to what extent would you like it to be supplemented with 
computer based simulation of a TDG? 


1 = not satisfactorily 

2 

3 

1 4 = satisfactorily 

5 

6 

7 = exceptionally 







well 


Thank you very much for participating in the study. 
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APPENDIX I - TDS SURVEY (EXPERIMENT 2) 


Pilot Survey 
- TDS Session 1 - 
Pre-questionnaire 

Please fill in the following questionnaire. Answer all questions as objectively as you can. All 

information will be held confidential. 

If you need to expand any answer please use the reverse side of your paper sheet. 

1) Were you required to use computer-based simulations as part of your training in the past? NO / YES (circle 
one) 

a) NO. 

b) If YES: What type simulations have you used and what skills were they used to train? 
i) Tactical Level (Marksmanship trainer, ship driving simulator, Language Trainer, etc.) 

(1) Description: 


(2) Estimate total number of hours you used it:_ 

(3) To gain competency, this amount of time spent was: (Circle one) 



ii) Operational Level (OneSAF, JSAF, JCATS, Common Operational Picture of the Battlespace, etc.) 
Description: 


Estimate total number of hours you used it:_ 

(1) To gain competency, this amount of time spent was: (Circle one) 

1 = Not ^ 

2 3 4 =Average 5 6 7 = Excellent 

Satisfactory 

2) Do you have experience playing commercial video games? NO / YES (circle one) 
a) NO. 


129 





b) If YES, What kind of video games do you 

week, month or year in each iine) 

i) First person shooter. (Stand Alone or Online) 

year 

ii) Flight Simulation. (Stand Alone or Online) 
year 

iii) Racing. (Stand Alone or Online) 

year 

iv) Sports (Football, etc.) (Stand Alone or Online) 

year 

v) Puzzle, Card, Board (Stand Alone or Online) 

year 

vi) Strategy (Stand Alone or Online) 

year 

vii) Adventure, Fantasy (Stand Alone or Online) 

year 

viii) Arcade (Stand Alone or Online) 

year 

ix) Other (Stand Alone or Online) 

year 


play? (please answer all that apply to you; circle day. 


Approximately 

hours ner day / week / month / 

Approximately 

hours per day / week / month / 

Approximately 

hours uer day / week / month / 

Approximately 

hours per day / week / month / 

Approximately 

hours per day / week / month / 

Approximately 

hours per day / week / month / 

Approximately 

hours per day / week / month / 

Approximately 

hours per day / week / month / 

Approximately 

hours per day / week / month / 


3) Do you own a personal computer? NO / YES (circle one) 

a) NO. 

h) If YES, how often do you use it? Approximately_hours per day / week / month / year 

4) What hand do you use to operate a computer mouse? LEFT / RIGHT / I am good with either (circle 


5) How often do you use the following other computer applications? (please answer all that apply to you; circle 


hours per day / week / month / year 
hours per day / week / month / year 
hours per day / week / month / year 
hours per day / week / month / year 
hours per day / week / month / year 
hours per day / week / month / year 
hours per day / week / month / year 


c, month, or year in each 

line) 

i) 

E-mail: 

Approximately 

ii) 

Browse Web: 

Approximately 

iii) 

IMovie: 

Approximately 

iv) 

ITunes: 

Approximately 

V) 

Excel: 

Approximately 

Vi) 

Word: 

Approximately 

vii) 

Power point: 

Approximately 


6) When you think about your own knowledge of infantry tactics, techniques, and procedures, how would you 
rank them? (please circle NA or one number between 1 and 7 in each line) 


# 

Knowledge and skills: 

Your current level 

poor 

Excellent 

1. 

Familiarity with a five paragraph order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Employ Machine guns in support of Offensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 
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3. 

Employ Machine guns in support of Defensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Infantry Offensive Operations in Field 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Infantry Defensive Operations in Field 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Tactical Movement On Objective 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Fire and Maneuver 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

Infantry Attack in Urban Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Infantry Defense in Urban Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Issue an Order 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Conduct Tactical Movement in the Offense 

NA 

1 

2 

3 

4 

5 

6 

7 


1) Subject# = (your code name):_ 

2) Date:_ Time: 

3) Year of birth: _ 

4) Country of Citizenship:_ 

5) Service component: (Circle one): 


Army 

Navy 

Air Force 

Marine Corps 

Coast Guard 

Civilian 


6) Briefly describe your Primary MOS, (Example: Infantry, Surface Warfare, communications, etc.): 


7) NPS Curriculum:_ 

8) Your Rank: _ 

9) Time in Service:_years_months 

Thank you very much for participating in the study. 
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Pilot Study Survey 


- TDS Session 2 - 
Platoon Leader Questionnaire 

Please fill in the following questionnaire. Answer all questions as objectively as you can. 

All information will be held confidential. 

If you need to expand any answer please use the reverse side of your paper sheet. 


Participant # = (Your code name):_ 

Date:_ Time:_ 

1) What training group were you in: (Circle One) 

a. Offense 

b. Defense 

2) For each person (including yourself) give a score between 0 and 100 which estimates the degree to which that 
person was the ‘leader’ in the last training session. (The four scores must add to 100). 

( For example , if no clear leader emerged, then each should have a score of 25. If Person X played the major role 
in this regard, then Person X would have a score of say 80, and the other 2 people, a score of 10 each. A score 
of 0 means that the corresponding person was in no way a leader, whereas a score of 100 means that the 
corresponding person was the clear and only leader.) 


Person (code name): 

Score/100: 









Total Score: 

100 


3) Think about a previous time when you enjoyed working together in a group. To what extent have you enjoyed 
the group experience just now? 


I enjoyed the group experience just now (please check one number between 1 and 7): 


1. Not at all 


It was nothing like the previous experience of working in a group. 

2. 



3. 



4. 



5. 



6. 



7. Very much so. 


I enjoyed it just as much as the previous experience of working in a group. 


4) Think back now about the session that you have just completed. For example, to what extent in your 
imagination can you move around that terrain (battlefield) now? 


I can think myself back on that terrain/battlefield (please check one number between 1 and 7): 


1. Not at all 


I cannot imagine now at all being on that terrain/battlefield now. 

2. 



3. 



4. 
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5. 



6. 



7. Very much so. 


I can strongly imagine being back on that terrain/battlefield now. 


5) In the last session, to what extent did you have the sense of the other three people being together with you ? 


In the last session, I had a sense that the other three people were with me (please check one number between 1 
and 7): 


1 1. Not at all 


1 I did not have a sense that the other three people were with me. | 




3. 



4. 



5. 



6. 



7. Very much so. 


The other three people were very much with me. 


6) Who did most of the talking? Give a score to each person between 0 and 100, where the four scores add to 
100. (A person would be given a score of near 100 only if they did almost all the talking. They would be given a 
score of near 0 if they did almost no talking). 

The extent to which each person did most of the talking was: 


Person (code name): 

Score/100: 









Total Score: 

100 


7) Overall, how cooperative was each of the other three people (please check one value between 1 and 7 for 
each person). 


1 Person (code name) 





1 1. Not at all 




S/he was not cooperative at all 
















5. 





6. 





7. Very much so. 




s/he was very cooperative 


8) Think back now about past session, what were good tactical decision and procedures done by your team? 
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9) Think back now about past session, what were bad tactical decision and procedures done by your team? 


10) Think back now about past session, what were good tactical decision and procedures done by other team? 


11) Think back now about past session, what were bad tactical decision and procedures done by other team? 
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12) What can you say about your SQUAD LEADERS on the following points, where Poor (1) - Excellent (7). 
(please circle NA or one number between 1 and 7 in each line) 

a) Squad Leader #1, Participant’s code name:_ 


# 

Knowledge and skills: 

poor 

Excellent 

Your current level 


1. 

Understood Operations Order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Identified Enemy Most Probably Course of Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created a Scheme of Maneuver (SOM) 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Considers Adjacent Units 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Created a Fire Support Plan (FSP) 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Identified Secondary MPCOA 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Considered the following in Mission Planning 
Process: 









7.a 

- Mission 

NA 

1 

2 

3 

4 

5 

6 

7 

7.b 

- Enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

7.C 

- Troops Available 

NA 

1 

2 

3 

4 

5 

6 

7 

7.d 

- Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

7.e 

- Time 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

Identified Enemy of Center of Gravity and 

Attacks It. 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Adjusted to Enemy Situation 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Took Charge 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Maintained Oversight of All Subordinate Units 

NA 

1 

2 

3 

4 

5 

6 

7 

12. 

Communicated Flis Plan/Intent to Subordinates 

NA 

1 

2 

3 

4 

5 

6 

7 

13. 

Ensured that Subordinates what they had been 
told 

NA 

1 

2 

3 

4 

5 

6 

7 

14. 

Used deception and creativity to outwit the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

15. 

Deviated from doctrine when necessary 

NA 

1 

2 

3 

4 

5 

6 

7 

16. 

Enemy Location 

NA 

1 

2 

3 

4 

.. 

5 

6 
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# 

Knowledge and skills: 

poor 

Excellent 

Your current level 


17. 

Size of Enemy Element 

NA 

1 

2 

3 

4 

5 

6 

7 

18. 

Direction of Enemy Movement 

NA 

1 

2 

3 

4 

5 

6 

7 

19. 

Understanding of Battle Space Geometry 

NA 

1 

2 

3 

4 

5 

6 

7 


b) Squad Leader #2, Participant’s code name: 


# 

Knowledge and skills: 

Your current level 

poor 

Excellent 

1. 

Understood Operations Order 

NA 

1 

2 1 3 

4 

5 

6 

7 

2. 

Identified Enemy Most Probably Course of Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created a Scheme of Maneuver (SOM) 

NA 

1 

2 1 3 

4 

5 

6 

7 

4. 

Considers Adjacent Units 

NA 

1 

2 1 3 

4 

5 

6 

7 

5. 

Created a Fire Support Plan (ESP) 

NA 

1 

2 1 3 

4 

5 

6 

7 

6. 

Identified Secondary MPCOA 

NA 

1 

2 1 3 

4 

5 

6 

7 

7. 

Considered the following in Mission Planning 
Process: 









7.a 

- Mission 

NA 

1 

2 1 3 

4 

5 

6 

7 

7.b 

- Enemy 

NA 

1 

2 1 3 

4 

5 

6 

7 

7.C 

- Troops Available 

NA 

1 

2 1 3 

4 

5 

6 

7 

7.d 

- Terrain 

NA 

1 

2 1 3 

4 

5 

6 

7 

7.e 

- Time 

NA 

1 

2 1 3 

4 

5 

6 

7 

8. 

Identified Enemy of Center of Gravity and 

Attacks It. 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Adjusted to Enemy Situation 

NA 

1 

2 1 3 

4 

5 

6 

7 

10. 

11. 

Took Charge 

NA 

1 

2 1 3 

4 

5 

6 

7 

Maintained Oversight of All Subordinate Units 

NA 

1 

2 1 3 

4 

5 

6 

7 

12. 

Communicated His Plan/Intent to Subordinates 

NA 

1 

2 1 3 

4 

5 

6 

7 

13. 

Ensured that Subordinates what they had been 
told 

NA 

1 

2 

3 

4 

5 

6 

7 

14. 

Used deception and creativity to outwit the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

15. 

Deviated from doctrine when necessary 

NA 

1 

2 1 3 

4 

5 

6 

7 

16. 

Enemy Location 

NA 

1 

2 1 3 

4 

5 

6 

7 

17. 

Size of Enemy Element 

NA 

1 

2 1 3 

4 

5 

6 

7 
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Knowledge and skills: 

poor 

Excellent 

Your current level 

18. 

Direction of Enemy Movement 

NA 

1 1 

2 

3 4 

5 1 6 1 7 1 

19. 

Understanding of Battle Space Geometry 

NA 

1 1 

2 

3 4 

5 1 6 1 7 1 


c) Squad Leader #3, Participant’s code name: 


# 

Knowledge and skills: 

Your current level 

poor 

Excellent 

1. 

2. 

Understood Operations Order 

NA 

1 

2 1 3 

4 

5 1 6 

7 

Identified Enemy Most Probably Course of Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created a Scheme of Maneuver (SOM) 

NA 

1 

2 1 3 

4 

5 1 6 

7 

4. 

Considers Adjacent Units 

NA 

1 

2 1 3 

4 

5 1 6 

7 

5. 

6. 

Created a Fire Support Plan (ESP) 

NA 

1 

2 1 3 

4 

5 1 6 

7 

Identified Secondary MPCOA 

NA 

1 

2 1 3 

4 

5 1 6 

7 

7. 

Considered the following in Mission Planning 
Process 









7.a 

- Mission 

NA 

1 

2 1 3 

4 

5 1 6 

7 

7.b 

- Enemy 

NA 

1 

2 1 3 

4 

5 1 6 

7 

7.C 

- Troops Available 

NA 

1 

2 1 3 

4 

5 1 6 

7 

7.d 

- Terrain 

NA 

1 

2 1 3 

4 

5 1 6 

7 

7.e 

- Time 

NA 

1 

2 1 3 

4 

5 1 6 

7 

8. 

9. 

Identified Enemy of Center of Gravity and 

Attacks It. 

NA 

1 

2 

3 

4 

5 

6 

7 

Adjusted to Enemy Situation 

NA 

1 

2 1 3 

4 

5 1 6 

7 

10. 

Took Charge 

NA 

1 

2 1 3 

4 

5 1 6 

7 

11. 

Maintained Oversight of All Subordinate Units 

NA 

L" 

2 1 3 

4 

5 1 6 

7 

12. 

Communicated His Plan/Intent to Subordinates 

NA 

1 

2 1 3 

4 

5 1 6 

7 

13. 

Ensured that Subordinates what they had been 
told 

NA 

1 

2 

3 

4 

5 

6 

7 

14. 

15. 

Used deception and creativity to outwit the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

Deviated from doctrine when necessary 

NA 

1 

2 1 3 

4 

5 1 6 

7 

16. 

17. 

Enemy Location 

NA 

1 

2 1 3 

4 

5 1 6 

7 

Size of Enemy Element 

NA 

1 

2 1 3 

4 

5 1 6 

7 

18. 

Direction of Enemy Movement 

NA 

1 

2 1 3 

4 

5 1 6 

7 
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# Knowledge and skills: Your cunent level 


poor 

Excellent 


19. 

Understanding of Battle Space Geometry 

NA 

1 

2 

3 

4 

5 

6 

7 

d) YOURSELF on the following criteria where Poor (1) - Excellent (7). (please circle NA or one 

number between 1 and 7 in each line) 

# 

Knowledge and skills: 

Your current level 

poor 

Excellent 

1. 

Understood Operations Order 

NA 

1 i 2 

3 

4 

5 

6 

7 

2. 

Identified Enemy Most Probably Course of Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created a Scheme of Maneuver (SOM) 

NA 

1 ! 2 

3 

4 

5 

6 

7 

4. 

Considers Adjacent Units 

NA 

1 2 

3 

4 

5 

6 

7 

5. 

Created a Fire Support Plan (FSP) 

NA 

1 i 2 

3 

4 

5 

6 

7 

6. 

Identified Secondary MPCOA 

NA 

1 1 2 

3 

4 

5 

6 

7 

7. 

Considered the following in Mission Planning 
Process: 









7.a 

- Mission 

NA 

1 2 

3 

4 

5 

6 

7 

7.b 

- Enemy 

NA 

1 1 2 

3 

4 

5 

6 

7 

7.C 

- Troops Available 

NA 

1 1 2 

3 

4 

5 

6 

7 

7.d 

- Terrain 

NA 

1 2 

3 

4 

5 

6 

7 

7.e 

- Time 

NA 

1 1 2 

3 

4 

5 

6 

7 

8. 

Identified Enemy of Center of Gravity and 

Attacks It. 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Adjusted to Enemy Situation 

NA 

1 ! 2 

3 

4 

5 

6 

7 

10. 

Took Charge 

NA 

1 1 2 

3 

4 

5 

6 

7 

11. 

Maintained Oversight of All Subordinate Units 

NA 

1 1 2 

3 

4 

5 

6 

7 

12. 

Communicated Flis Plan/Intent to Subordinates 

NA 

1 I 2 

3 

4 

5 

6 

7 

13. 

Ensured that Subordinates what they had been 
told 

NA 

1 

2 

3 

4 

5 

6 

7 

14. 

Used deception and creativity to outwit the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

15. 

Deviated from doctrine when necessary 

NA 

1 1 2 

3 

4 

5 

6 

... 

7 

16. 

Enemy Location 

NA 

1 i 2 

3 

4 

5 

. .._ 

6 

7 

17. 

Size of Enemy Element 

NA 

1 1 2 

3 

4 

5 

6 

7 

18. 

Direction of Enemy Movement 

NA 

1 1 2 

3 

4 

5 

6 

7 
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# Knowledge and skills: 

19. Understanding of Battle Space Geometry 


Your current level 


poor 

Excellent 


NA 

1 

2 

3 

4 

5 

6 1 


13) If you did not have enough time to become familiar with the infantry TTPs of the offense or defense 
(whichever you participated in), how much more time do you believe you would need? 


14) How satisfied were you with your overall performance while using the CCM TDS: 


l = Not 
Satisfied 


2 


3 


4 =Average 


5 


6 


7 = Extremely 


Thanks very much for participating! 


NOTE: 
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Pilot Study Survey 


- TDS FINAL - 


Please fill in the following questionnaire. Answer all questions as objectively as you can. 

All information will be held confidential. 

If you need to expand any answer please use the reverse side of your paper sheet. 

1) Which of the following items/drills did you exercise at least once during the TDS sessions? (Circle all that 
apply from the list below.) 


# 

Knowledge and skills: 

1. 

Familiarity with a five paragraph order 

2. 

Employ Machine guns in support of Offensive Operations 

3. 

Employ Machine guns in support of Defensive Operations 

4. 

Infantry Offensive Operations in Field 

5. 

Infantry Defensive Operations in Field 

6. 

Tactical Movement On Objective 

7. 

Fire and Maneuver 

8. 

Infantry Attack in Urban Terrain 

9. 

Infantry Defense in Urban Terrain 

10. 

Issue an Order 

11. 

Conduct Tactical Movement in the Offense 


2) Of the actions that you chose in the previous question (Question 1), with which did you have the most 
difficulty ? 

(circle three) 


# 

Knowledge and skills: 

1. 

Familiarity with a five paragraph order 

2. 

Employ Machine guns in support of Offensive Operations 

3. 

Employ Machine guns in support of Defensive Operations 

4. 

Infantry Offensive Operations in Field 

5. 

Infantry Defensive Operations in Field 

6. 

Tactical Movement On Objective 
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# 

Knowledge and skills: 

7. 

Fire and Maneuver 

8. 

Infantry Attack in Urban Terrain 

9. 

Infantry Defense in Urban Terrain 

10. 

Issue an Order 

11. 

Conduct Tactical Movement in the Offense 


3) Of the actions that you chose in the Question 1, which do you feel you improved upon most during the TDG 

sessions? (circle three) 


# 

Knowledge and skills: 

1. 

Familiarity with a five paragraph order 

2. 

Employ Machine guns in support of Offensive Operations 

3. 

Employ Machine guns in support of Defensive Operations 

4. 

Infantry Offensive Operations in Field 

5. 

Infantry Defensive Operations in Field 

6. 

Tactical Movement On Objective 

7. 

Fire and Maneuver 

8. 

Infantry Attack in Urban Terrain 

9. 

Infantry Defense in Urban Terrain 

10. 

Issue an Order 

11. 

Conduct Tactical Movement in the Offense 


4) When you think about your own knowledge of infantry tactics, techniques, and procedures, how would you 
rank them? (please circle NA or one number between 1 and 7 in each line) 


# 

Knowledge and skills: 

Your current level 

Poor 

Excellent 

1. 

Familiarity with a five paragraph order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Employ Machine guns in support of Offensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Employ Machine guns in support of Defensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Infantry Offensive Operations in Field 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Infantry Defensive Operations in Field 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Tactical Movement On Objective 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Fire and Maneuver 

NA 

1 

2 

_1 

3 

_ 

4 

5 

6 

7 
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# 

Knowledge and skills: 

Poor 

Excellent 

Your current level 


8. 

Infantry Attack in Urban Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Infantry Defense in Urban Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Issue an Order 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Conduct Tactical Movement in the Offense 

NA 

1 

2 

3 

4 

5 

6 

7 


5) When you think about your own knowledge of infantry tactics, techniques, and procedures, how much did 
you improve in each? (please circle NA or one number between 1 and 7 in each line) 


# 

Knowledge and skills: Your improvement 

They did not 

Improved 

Improve at all 

greatly 

1 

Familiarity with a five paragraph order 

NA 

1 

2 

3 

4 

5 

6 

7 

2 

Employ Machine guns in support of Offensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

3 

Employ Machine guns in support of Defensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

4 

Infantry Offensive Operations in Field 

NA 

1 

2 

3 

4 

5 

6 

7 

5 

Infantry Defensive Operations in Field 

NA 

1 

2 

3 

4 

5 

6 

7 

6 

Tactical Movement On Objective 

NA 

1 

2 

3 

4 

5 

6 

7 

7 

Fire and Maneuver 

NA 

1 

2 

3 

4 

5 

6 

7 

8 

Infantry Attack in Urban Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

9 

Infantry Defense in Urban Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

1 

Issue an Order 

NA 

1 

2 

3 

4 

5 

6 

7 

1 

Conduct Tactical Movement in the Offense 

NA 

1 

2 

3 

4 

5 

6 

7 


6) How would you rank your ability to navigate through the virtual world and use the controls in simulation? 


1. 

NAVIGATION/MOVING 
THROUGH THE TERRAIN: 

1 = 

POOR 

2 

3 

4 =AVERAGE 

5 

6 

7 = 

EXCELLENT 

2. 

OTHER CONTROLS: 

1 = 

POOR 

2 

3 

4 =AVERAGE 

5 

6 

7 = 

EXCELLENT 


7) Rate the simulation ability to provide you with good visual and audio cues so that you can identify all threats 
during the mission: 


1. 

Visual cues: 

1 = poor 

2 

3 

4 = average 

5 

6 

7 = excellent 

2. 

Audio cues: 

1 = poor 

2 

3 

4 = average 

5 

6 

7 = excellent 
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8) Rate the following statements (circle one number for each statement): 


1. 

“During the session with computer simulation, I 
felt like I was taking part in training.” 


1 = strongly 
disagree 

2 

3 

4 = 

neutral 

5 

6 

7 = 

Strongly 

agree 

2. 

“During the session with computer simulation, I 
felt like my actions in virtual world had no 
consequences to the others and to the virtual 
world.” 


1 = strongly 
disagree 

2 

3 

4 = 

neutral 

5 

6 

7 = 

Strongly 

agree 

3. 

“During the session with computer simulation, I 
felt like I was part of the group working together.” 


1 = strongly 
disagree 

2 

3 

4 = 

neutral 

~5' 

6 

7 = 

Strongly 

agree 

4. 

“During the session with computer simulation, I 
felt like I was playing a game.” 


1 = strongly 
disagree 

2 

3 

4 = 

neutral 

5 

6 

7 = 

Strongly 

agree 

5. 

“During the session with computer simulation, I 
felt isolated from the others.” 


1 = strongly 
disagree 

2 

3 

4 = 

neutral 

5 

6 

7 = 

Strongly 

agree 


9) In your opinion what was the overall quality of training in computer based training simulation? (circle one) 


1 = not satisfactory 2 3 4 = average 5 6 7 = excellent 


10) Throughout all the TDS sessions, please rate the quality of your tactical decisions, (circle one number) 


1 = poor 

2 

3 

4 = satisfactory 

5 

6 

7 = excellent 


11) What were your expectations regarding the ability of simulation to train you in the previous set of skills 
mentioned in Questions 1, 2, and 3? 


1 = low 2 3 4 = average 5 6 7 = high 


12) How well did the use of simulation meet your expectations regarding training value? 


1 = not satisfactorily 

2 

3 

4 = satisfactorily 

5 

6 

7 = exceptionally 
well 


13) Circle one number that best reflects your opinion: 


When you think about your future training 

1 = not 

2 

3 

4 = 

5 

6 

7 = very 

needs, to what extent would you like it to be 

at all 



moderately 



much 

supplemented with computer based 








simulations like CCM? 









Thank you very much for participating in the study. 
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APPENDIX J - TDG/TDS SCHEDULE (EXPERIMENT 2) 


TDG 

- APPROXIMATED TBS Design - 


Phase I - Pre-Training (Individual) 

SETTING: Classroom/Lab (Watkins 212A & MOCAP Lab) 

METHOD OF INSTRUCTION: TDG 

00-10 TIME: 10 Minutes - Initial Long Pre-Questionnaire 

SCENARIO: 1-la- Town(a) 

10-30 TIME: 20 Minutes - Conduct of the TDG 
TIME: Minutes - What is a TDG? 

-Planning Sequence 

1. Read and Understand Operation Order 

2. Estimate Enemy activity 

3. Plan your response 

4. What-if your plan. 

5. Move troops to attack/defense positions 

6. Conduct Offensive/Defensive operations 

7. Conduct After-Action Review 

-Offensive Tactics Training (OPFOR) 

1. Support by Fire Positions 

2. Distribution of Forces 

3. Exposure (use terrain to mask movement and positions) 

4. Use Smoke to cover movement. 

5. Create and Use a Fire Support Plan 

6. Don’t give conflicting orders (ex. “Dig in” and “Move Fast”) 

7. Maneuver Warfare 

a) Fire and Maneuver [SAM-K (Suppress, Assess, Maneuver, Kill)] (I cover you, you cover me during movements). 

b) De-escalation of the Fire support upon approach 

i. Artillery 

ii. BN mortars 

iii. CO mortars 

iv. Medium machine guns 

V. Squad automatic weapons 

vi. M-16 

vii. Grenades 

8. Tactical movement (moving fast or moving covert) 

9. LongAJnsupported movements are bad 

10. Use of the reserve 

- Defensive Tactics Training (MARINES) 

1. Dig in when possible 

2. Envision Enemy’s attack based on the OpOrd 

3. Select the ground where you can stop the enemy attack 

4. Select troop battle positions based on your expectation 

5. Ensure interlocking Sectors of Fire on this ground 

6. Envision the enemy’s reaction to this initial contact 

7. Have a plan to counter the enemy’s reaction to initial contact 

8. Give key weapons primary and secondary positions of expected enemy action/reaction 

9. Use Rifle Squads to cover the flanks of your key weapons 

10. Use covered movements between primary and supplemental positions (some units may not move) 

11. Control of Fires 

12. Be flexible, (i.e.. Don’t stick to your plan if it is not working.) 

13. Use primary, alternate, supplemental positions, a reserve. 

NOTE: Platoons and squads are not large enough to designate a static reserve. One unit should be designated to break contact 
and act as the reserve if needed. 

14. Establish a withdrawal plan if you need to evacuate the area, (route, sequence, 
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30-40 Conducting a 10 minute TDG introduction 
00-02 Division Of The Teams Into 2 Groups 
02-04 Passing out The Material For TDG 
04-10 Explanation of what Each Person Is Supposed To 
40-45 TIME: 5 Minutes - After Action 
45-50 TIME: 5 Minutes - Post Questionnaire 

Phase II - Tactics Training - Defense 
(Team) 

SCENARIO: 3-Ia - Town (b) 


00-05 TIME: 5 Minutes Instructor Introduction with Reading of OpOrder 

05-15 TIME: 10 Minutes 

TIME: 5 Minutes Group Discussion 

TIME: 5 Minutes Individual Student Planning 
-Mission Enemy Terrain Troops - Time (METT-T) 

-conduct map recon 
-estimate of situation 
-sketch tactical plan on map 

15-25 TIME: 10 Minutes Internal Team discussion on WHY each individual chose his COA 

25-49 TIME: 24 Minutes Team Leader Picked and Briefs Overall Plan (Instructors may pause for critical discussion points) 

TIME 12: Team 1 Briefs 
Time 12: Team 2 Briefs 

49-54 TIME: 5 Minutes - Individuals Write down the good/bad points of your team’s and the other team’s plan 
54-59 TIME: 5 Minutes - Post-Questionnaire with After Action Section 

(Subject Matter Experts (SMEs) Provide Evaluation on Tactical Decision-Making and 
Platoon Level Leadership) 

59-65 TIME: 6 Minutes - Phase II After-Action Review 


Source of Time Information: J.D. Deforest combined with DVD of Experiment with Dr. Sadagic 

Phase III - Tactics Training - Offense 
(Team Internal Role Shift) 

SCENARIO: 3-Ia- Town (b) 


00-05 TIME: 5 Minutes Instructor Introduction with Reading of OpOrder 

05-15 TIME: 10 Minutes 

TIME: 5 Minutes Group Discussion 

TIME: 5 Minutes Individual Student Planning 
-Mission Enemy Terrain Troops - Time (METT-T) 

-conduct map recon 
-estimate of situation 
-sketch tactical plan on map 

15-25 TIME: 10 Minutes Internal Team discussion on WHY each individual chose his COA 

25-49 TIME: 24 Minutes Team Leader Picked and Briefs Overall Plan (Instructors may pause for critical discussion points) 

TIME 12: Team 1 Briefs 
Time 12: Team 2 Briefs 

49-54 TIME: 5 Minutes - Individuals Write down the good/bad points of your team’s and the other team’s plan 

54-59 TIME: 5 Minutes - Post-Questionnaire with After Action Section 

(Subject Matter Experts (SMEs) Provide Evaluation on Tactical Decision-Making and 
Platoon Level Leadership) 

59-65 TIME: 6 Minutes - Phase II After-Action Review 

Source of Time Information: J.D. Deforest combined with DVD of Experiment with Dr. Sadagic 

Phase IV - Tactics Training - Defense 
(Team Internal Role Shift) 

SCENARIO: 2-la - Airfield 


00-05 TIME: 5 Minutes Instructor Introduction with Reading of OpOrder 
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05-15 TIME: 10 Minutes 

TIME: 5 Minutes Group Discussion 

TIME: 5 Minutes Individual Student Planning 
-Mission Enemy Terrain Troops - Time (METT-T) 

-conduct map recon 
-estimate of situation 
-sketch tactical plan on map 

15-25 TIME: 10 Minutes Internal Team discussion on WHY each individual chose his COA 

25-49 TIME: 24 Minutes Team Leader Picked and Briefs Overall Plan (Instructors may pause for critical discussion points) 

TIME 12: Team 1 Briefs 
Time 12: Team 2 Briefs 

49-54 TIME: 5 Minutes - Individuals Write down the good/bad points of your team’s and the other team’s plan 

54-59 TIME: 5 Minutes - Post-Questionnaire with After Action Section 

(Subject Matter Experts (SMEs) Provide Evaluation on Tactical Decision-Making and 
Platoon Level Leadership) 

59-65 TIME: 6 Minutes - Phase II After-Action Review 


Source of Time Information: J.D. Deforest combined with DVD of Experiment with Dr. Sadagic 

Phase V - Tactics Training/Force on Force 
(Team Internal Role Shift) 

SCENARIO: 6-la-Farm 

00-05 TIME: 5 Minutes Instructor Introduction with Reading of OpOrder 

05-15 TIME: 10 Minutes 

TIME: 5 Minutes Group Discussion 

TIME: 5 Minutes Individual Student Planning 
-Mission Enemy Terrain Troops - Time (METT-T) 

-conduct map recon 
-estimate of situation 
-sketch tactical plan on map 

15-25 TIME: 10 Minutes Internal Team discussion on WHY each individual chose his COA 

25-49 TIME: 24 Minutes Team Leader Picked and Briefs Overall Plan (Instructors may pause for critical discussion points) 

TIME 12: Team 1 Briefs 
Time 12: Team 2 Briefs 

49-54 TIME: 5 Minutes - Individuals Write down the good/bad points of your team’s and the other team’s plan 

54-59 TIME: 5 Minutes - Post-Questionnaire with After Action Section 

(Subject Matter Experts (SMEs) Provide Evaluation on Tactical Decision-Making and 
Platoon Level Leadership) 

59-65 TIME: 6 Minutes - Phase II After-Action Review 


Source of Time Information: J.D. Deforest combined with DVD of Experiment with Dr. Sadagic 
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TDS 

- Pilot Study Design - 


Phase I - Pre-Training 
(Individual Intro to Tactics) 

SETTING: Classroom/Lab (Watkins 212A) 

METHOD OF INSTRUCTION: Lecture with PowerPoint Presentation. CCM. 

00-10 TIME: 10 Minutes - Initial Long Pre-Questionnaire 
10-15 TIME: 5 Minutes - Instruction via Lecture & PPT. 

-Show Opening CCM Screen 

-Describe the three entry points (Command, Editor, Boot Camp) 

-Go into Command 

-Operation of the simulation 

1. Left Click = Highlight Unit, Right Click = Issue Orders 

2. Ambush 30m/Defend 200m 

3. Firing/Range indicator 

4. # on Buildings = # indicates Levels 

5. Navigation - arrows/map 

6. To get Elevation Data - Right Click on terrain 

7. Space Bar - Leadership Circle 

15-35 TIME: 20 Minutes (10 Minutes OPFOR/10 Minutes MARINES) 

-Planning Sequence 

1. Read and Understand Operation Order 

2. Estimate Enemy activity 

3. Plan your response 

4. What-if your plan. 

5. Move troops to attack/defense positions 

6. Conduct Offensive/Defensive operations 

7. Conduct After-Action Review 

-Offensive Tactics Training (OPFOR) 

1. Support by Fire Positions 

2. Distribution of Forces 

3. Exposure (use terrain to mask movement and positions) 

4. Use Smoke to cover movement. 

5. Create and Use a Fire Support Plan 

6. Don’t give conflicting orders (ex. “Dig in” and “Move Fast”) 

7. Maneuver Warfare 

a) Fire and Maneuver [SAM-K (Suppress, Assess, Maneuver, Kill)] (I cover you, you cover me during movements). 

b) De-escalation of the Fire support upon approach 

i. Artillery 

ii. BN mortars 

iii. CO mortars 

iv. Medium machine guns 

V. Squad automatic weapons 

vi. M-16 

vii. Grenades 

8. Tactical movement (moving fast or moving covert) 

9. LongAJnsupported movements are bad 

10. Use of the reserve 

- Defensive Tactics Training (MARINES) 

1. Dig in when possible 

2. Envision Enemy’s attack based on the OpOrd 

3. Select the ground where you can stop the enemy attack 

4. Select troop battle positions based on your expectation 

5. Ensure interlocking Sectors of Fire on this ground 

6. Envision the enemy’s reaction to this initial contact 

7. Have a plan to counter the enemy’s reaction to initial contact 
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8. Give key weapons primary and secondary positions of expected enemy action/reaction 

9. Use Rifle Squads to cover the flanks of your key weapons 

10. Use covered movements between primary and supplemental positions (some units may not move) 

11. Control of Fires 

12. Be flexible, (i.e., Don’t stick to your plan if it is not working.) 

13. Use primary, alternate, supplemental positions, a reserve. 

NOTE: Platoons and squads are not large enough to designate a static reserve. One unit should be designated to break contact and act 
as the reserve if needed. 

14. Establish a withdrawal plan if you need to evacuate the area, (route, sequence, 

SCENARIO; 1-la - Town (a) as OPFOR for Offense/Marines for Defense 
(Researchers Available for Questions/Flelp) 

Individual Hands-On Session 
35-37 TIME: 2 Minutes - Placement of Troops 
37-47 TIME: 10 Minutes - Offensive Training 
47-57 TIME: 10 Minutes - Defensive Training 
57-62 TIME: 5 Minutes - After Action 
62-67 TIME: 5 Minutes - Post Questionnaire 


Phase II - Team Tactics 
(Team Offense or Team Defense) 

SETTING: Classroom/Lab (Watkins 2I2A & MOCAP Lab) 

METHOD OF INSTRUCTION: Lecture with CCM. 

00-10 TIME: 10 Minutes - Tactics Review from Phase I 
- REVIEW Offensive Tactics Training (5 Minutes) 

- REVIEW Defensive Tactics Training (5 Minutes) 

SCENARIO: 3-lb - Town (b) 

(OPFOR for Offense/Marines for Defense (Researchers Available for Questions/Help)) 

10-12 TIME: 2 Minutes - Issue Operations Order 
12-15 TIME: 3 Minute - Planning 
15-17 TIME: 2 Minutes - Troops Placement 
17-39 TIME: 22 Minutes - Team CCM Hands-On 

32-34 TIME: 2 Minutes - GRADE Evaluation of Situational Awareness 

39-44 TIME: 5 Minutes - Post-Questionnaire with After Action Section 

(Subject Matter Experts (SMEs) Provide Evaluation on Tactical Decision-Making and 
Platoon Level Leadership) 

44-49 TIME: 5 Minutes - Phase II After-Action Review 


Phase III - Team Tactics 

(Team Offense or Team Defense...Switch from Phase II) 

SETTING: Classroom/Lab (Watkins 212A & MOCAP Lab) 

METHOD OF INSTRUCTION: CCM. 


SCENARIO: 3-lb - Town (b) 

(OPFOR for Offense/Marines) 

05-07 TIME: 2 Minutes - Issue Operations Order 
07-10 TIME: 3 Minute - Planning 
10-12 TIME: 2 Minutes - Troops Placement 
12-56 TIME: 44 Minutes - Team CCM Hands-On 
32-34 TIME: 2 Minutes -GRADE 1 
44-46 TIME: 2 Minutes -GRADE 2 
56-61 TIME: 5 Minutes - Post-Questionnaire with After Action Section 

(Subject Matter Experts (SMEs) Provide Evaluation on Tactical Decision-Making and 
Platoon Level Leadership) 

61-66 TIME: 5 Minutes - Phase III After-Action Review 
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Phase IV - Team Tactics 

(Team Offense or Team Defense...Switch from Phase III) 

SETTING: Classroom/Lab (Watkins 212A & MOCAP Lab) 

METHOD OF INSTRUCTION: CCM. 

SCENARIO: 2-Ib - Airfield 

(OPFOR for Offense/Marines for Defense (Researchers Available for Questions/Help) 

05-07 TIME: 2 Minutes - Issue Operations Order 
07-10 TIME: 3 Minute - Planning 
10-12 TIME: 2 Minutes - Troops Placement 
12-56 TIME: 44 Minutes - Team CCM Hands-On 
32-34 TIME: 2 Minutes - GRADE 1 
44-46 TIME: 2 Minutes - GRADE 2 
56-61 TIME: 5 Minutes - Post-Questionnaire with After Action Section 

(Subject Matter Experts (SMEs) Provide Evaluation on Tactical Decision-Making and 
Platoon Level Leadership) 

61-66 TIME: 5 Minutes - Phase IV After-Action Review 


Phase V - Team Tactics 

(Team Offense or Team Defense...Switch from Phase IV) 

SETTING: Classroom/Lab (Watkins 212A & MOCAP Lab) 

METHOD OF INSTRUCTION: CCM. 


SCENARIO: 6-lb - Farm 

(OPFOR for Offense/Marines for Defense (Researchers Available for Questions/Help) 

05-07 TIME: 2 Minutes - Issue Operations Order 
07-10 TIME: 3 Minute - Planning 
10-12 TIME: 2 Minutes - Troops Placement 
12-56 TIME: 44 Minutes - Team CCM Hands-On 
32-34 TIME: 2 Minutes - GRADE 1 
44-46 TIME: 2 Minutes - GRADE 2 
56-61 TIME: 5 Minutes - Post-Questionnaire with After Action Section 

(Subject Matter Experts (SMEs) Provide Evaluation on Tactical Decision-Making and 
Platoon Level Leadership) 

61-66 TIME: 5 Minutes - Phase V After-Action Review 
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APPENDIX K - CCM OPERATION ORDERS (EXPERIMENT 2) 


SESSION 1 


Module 1 Fight 1-la OpOrder (Marines - DEFENSE) 

I. Situation: Enemy patrols, mostly squad size, have beeome inereasingly bolder as we 
have withdrawn (South). I think we ean expeet tentative eontaet here in the village this 
morning. Battalion withdraws (South) and repositions in order to turn over this seetor to 
allied forces. 

II. Mission: “A” Company guards the Battalion rear in order to prevent Enemy 
interference with our withdrawal. 

III. Execution: (1) Platoon is detached to Battalion. (1) platoon guards the town while (1) 
Platoon moves (South) to establish next rear guard Position. 

Eire Support: No Close Air Support. No Artillery. No Mortars. 

Tasks: 

1st Platoon: Main Effort. Guard the Company rear in order to prevent Enemy 
interference with our withdrawal. 

2nd Platoon: Detached. 

3rd Platoon: Move (South) and establish next rear guard Position in order to 
permit 1st Platoon to withdraw. 

Mortars: Priority of hires to 3rd Platoon. 

IV. Admin and Eogistics: Standard Operating Procedure. 

V. Command and Signal: Standard Operating Procedure. 
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SESSION 1 


Module 1 Fight 1-la OpOrder (Opposing Force (OPFOR) - OFFENSE) 

I. Situation; An Enemy squad with a machine gun holds the village. I think they missed 
their withdrawal and are now planning on fighting to the end. Battalion secures a 
Defensive Position 6k (East) in order to destroy an anticipated Enemy Counter-Attack. 

II. Mission: “E” Company clears the town in order to remove Enemy forces. 

III. Execution; While (1) Main Effort Platoon clears (South) through the village, (1) 
Platoon screens (East) of the village. (1) Platoon Reserve. 

Eire Support; No Close Air support. No Artillery. 

Tasks: 

1st Platoon; Main Effort. Clear (South) through the village lOT remove Enemy 
forces. 

2nd Platoon; Screen (East) of the village in order to identify and report Enemy 
activities. 

3rd Platoon; Reserve. Eollow in Trace of 1st Platoon. Be prepared to Cover Main 
Effort. 

Sniper Team; You are in Direct Support of the Main Effort. 

IV. Admin and Eogistics; Standard Operating Procedure. 

V. Command and Signal; Standard Operating Procedure. 
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SESSION 2 & 3 


Module 5 Fight 5-la OpOrder (OPFOR) 

I. Situation: Enemy patrols, mostly squad size, have beeome inereasingly bolder as we 
have withdrawn (West). I think we can expect tentative contact here in the village this 
morning. Battalion withdraws (West) and repositions in order to conduct a counter attack. 

II. Mission “E” Company guards the Battalion rear in order to prevent Enemy 
interference with our withdrawal. 

III. Execution (1) Platoon is detached to Battalion. (1) Platoon guards the town while (1) 
Platoon moves (West) to establish next rear guard Position. 

PS: No Close Air Support. No artillery. No Mortars. 

Tasks: 

1st Platoon: Main Effort. Guard the Company rear in order to prevent Enemy interference 
with our withdrawal. 

2nd Platoon: Detached. 

3rd Platoon: Move (West) and establish next rear guard Position in order to permit 1st 
Platoon to Withdraw. 

Mortars: Priority of Eire to 3rd Platoon. 

IV. Admin and Eogistics: Standard Operating Procedure. 

V. Command and Signal: Standard Operating Procedure. 
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SESSION 2 & 3 


Module 5 Fight 5-la OpOrder (MARINES-OFFENSE) 

I. Situation; An Enemy Squad with a Machine Gun holds the bank building, the last 
organized resistance in the village. I think they missed their withdrawal and are now 
planning on fighting to the end. Battalion secures a defensive position 6 kilometers 
(West) in order to stop an anticipated Enemy counter attack. 

II. Mission: “A” Company clears the town in order to free the road from Enemy control. 

III. Execution; While (1) Platoon clears (South), the Main effort Platoon clears (West) to 
the Rank. Reserve Platoon then clears the rest of town. 

PS: No Close Air Support. No Artillery. Mortars to 2nd Platoon. 

Tasks: 

1st Platoon; Main Effort. Clear (West) to the bank in order to free the road from Enemy 
control. 

2nd Platoon: Clear (South) across the river in order to free the road from Enemy control. 

3rd Platoon; RESERVE. Be Prepared To clear the town in order to free the road from 
Enemy control. 

Mortars; Priority of Eire is to 2nd Plat. 

IV. Admin and Eogistics; Standard Operating Procedure. 

V. Command and Signal; Standard Operating Procedure. 
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SESSION 4 


Module 2 Fight 2-la OpOrder (Marines - OFFENSE) 

I. Situation; Scattered Enemy units, maybe (20) soldiers, remain in the vieinity of the 
airfield. They may have a Heavy Maehine Gun, mortar, or Anti-Aireraft missile that 
could threaten our helicopters. The remainder of the battalion will fly into the airfield 
once the Anti-Aireraft threat is reduced in order to stage for follow-on operations. I think 
the Enemy will avoid eontaet with us and wait to target the helieopters. 

If. Mission: “A” Company elears the area around the airfield in order to prevent Enemy 
interference with air operations. 

Iff. Exeeution: While (1) Platoon bloeks the (North) aeeess road, (2) Platoons elear both 
sides of the airfield from (North) to (South). 

PS: Rotary Wing Close Air Support On-Call. No Artillery. Mortars with 1st Platoon. 
Tasks: 

1st Platoon; Main Effort. Clear (West) side of airfield in order to prevent Enemy 
interferenee with air operations. 

2nd Platoon; Clear (East) side of airfield in order to prevent Enemy interferenee 
with air operations. 

3rd Platoon; Bloek (North) approaeh to airfield in order to prevent Enemy 
Counter-Attack. 

Mortars: Priority of Eire to the Main Effort. 

IV. Admin and Eogistics: Standard Operating Proeedure. 

V. Command and Signal; Standard Operating Proeedure. 
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_ SESSION 4 _ 

Module 2 Fight 2-la OpOrder (Opposing Force (OpFor) - DEFENSE) 

I. Situation; An Enemy unit, possibly Company sized is loeated 10 kilometers to the 
(North West). They may be intending to use the airfield as a staging area for EOF. The 
remainder of our Battalion is located 15 kilometers to the (East). I think the Enemy will 
attempt to seize the airfield within the next (2) hours. 

If. Mission: “E” Company Controls the airfield in order to prevent Enemy use. 

Iff. Execution; (1) Platoon Clears (West) side of airfield, (1) Platoon Clears (East) side of 
airfield. (1) Platoon detached to Battalion. 

FS: No Close Air Support. No Artillery. Mortars to Main Effort. 

Tasks: 

1st Platoon; Main Effort. Clear the (West) side of airfield in order to prevent Enemy use. 
2nd Platoon; Clear (East) side of airfield in order to prevent Enemy use. 

3rd Platoon; Detached Mortars: Priority of Fire to Main Effort. 

IV. Admin and Eogistics: Standard Operating Procedure. 

V. Command and Signal; Standard Operating Procedure. 
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SESSION 5 


Module 6 Fight 6-la OpOrder (Marines - OFFENSE) 

I. Situation: An Enemy patrol has cut the road at the village, stranding a convoy farther 
(East). Enemy is believed to number (12) to (15) soldiers and have shoulder-fired 
missiles. The convoy reports additional Enemy movement in the hills to their (East), 
trapping them in place. I think the Enemy will focus on the road traffic and less on the 
overland approaches to the village. This main road links our Battalion, 4 kilometers 
(West), with 2nd Battalion, 17 kilometers (East). Tomorrow, Battalion moves (North) 
and secures river line in order to prevent Enemy crossings. 

If. Mission: “A” Company destroys the Enemy in the village in order to clear the road for 
convoy operations. 

Iff. Execution: (2) Platoons envelop the village from the (North). Heavy Machine Gun 
Squad provides SBE vicinity of the road. 

ES: Rotary Wing Close Air Support On call. Artillery On call. Mortars with Main 
Effort. 

Tasks: 

1st Platoon: Main Effort. Destroy the Enemy in the village in order to clear the road for 
convoy operations. 

2nd Platoon: RESERVE. Eollow in trace of 1st Platoon be prepared to continue Attack. 
3rd Platoon: Detached. 

Heavy Machine Gun Squad: Direct Support to Main Effort. 

Mortars: Priority of Eire to Main Effort. 

IV. Admin and Eogistics: Standard Operating Procedure. 

V. Command and Signal: Standard Operating Procedure. 
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SESSION 5 


Module 6 Fight 6-la OpOrder (Opposing Force (OpFor) - DEFENSE) 

I. Situation: Enemy Company has been sighted (West) of the town. I think the Enemy 
will attack us with a reinforced platoon here in town in order to gain control of this road 
network. Battalion secures this (East)-(West) Main Service Route in order to prevent 
Enemy interference with resupply convoys. 

II. Mission “E” Company secures the town in order to protect the Main Service Route 
from Enemy interference. 

III. Execution (1) Platoon secures the town. (1) Platoon secures the (South) road. (1) 
Platoon is detached. 

PS: No Close Air Support. No artillery. No Mortars. 

Tasks: 

1st Platoon: Main Effort. Secure the town in order to protect the Main Service Route. 

2nd Platoon: Detached. 

3rd Platoon: Move (South) and secure road sector in order to protect Main Service Route. 
Mortars: Priority of Eire is to 3rd Platoon. 

IV. Admin and Eogistics: Standard Operating Procedure. 

V. Command and Signal: Standard Operating Procedure. 
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APPENDIX L - TACTICS MEMORY SHEET (EXPERIMENT 2) 
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Offensive Tactics 


Use Terrain to Mask Mvmt Use of the Reserve 
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Defensive Tactics 


Select Battle Position 

Based on Expectation of Use Primary, Alternate, 

Enemy Advance Supplementary Positions 
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APPENDIX M - EXAMPLE GRADE SA SHEET (EXPERIMENT 2) 


Geographical Recall and Analysis of Data in the Environment (GRADE) 



Centrafia^l 


^ Tim^ / 2 "^ Time 
(choose one) 


L Session4 - 2-1 

Subject# "/Af 


[TaS^TDG 
(choose one) 


Steps 

la. Chooaa Cartainty 
Laval of Enamy Location 
Low 


lb. Pick Location on Map 

lc. lypa Percalvad Unit 
Siza (P-Platoon, S-Squad, T> 
Team) 

ld. ChooaejJi/fcJion of 
advance 


1e. Chooaa Location on Map 
2a. Chooaa Cartainty 
Laval of Friandly 
Location 

Low 


2b. Pick Location on Map 
2c. lype Perceived Unit 
Siza <P-Plataon, s^quad, T- 
Taam) 

2d. ChooafL^HtPtion of 
advance 


2a. Pick Location on Map 
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APPENDIX N - EVALUATION FORM (EXPERIMENT 2) 


TDS - Platoon Leader Leadership 









Evaluation Sheet 

Subject #: 









Team: 

Offense / Defense 









Evaluator: 










Date/Time: 









Likert Scale (1) Poor to (7) Excellent 



















# 

Knowledge and skills: 


. Platoon Leader's Current Level 




Poor 




Excellent 


PLANNER 









1 

Understood Operations Order 

NA 

1 

2 

3 

4 

5 

6 

7 

2 

Identified Enemy Most Probably Course of Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

■ 

3 

Created a Scheme of Maneuver (SOM) 

NA 

1 

2 

3 

4 

5 

6 

7 

4 

Considers Adjacent Units 

NA 

1 

2 

3 

4 

5 

6 

7 

5 

Created a Fire Support Plan (FSP) 

NA 

1 

2 

3 

4 

5 

6 

7 

6 

Identified Secondary MPCOA 

NA 

1 

2 

3 

4 

5 

6 

7 

7 

Considered the following in Mission Planning 
Process 

NA 

1 

2 

3 

4 

5 

6 

7 

8 

- Mission 

NA 

1 

2 

3 

4 

5 

6 

7 

9 

- Enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

10 

- Troops Available 

NA 

1 

2 

3 

4 

5 

6 

7 

11 

- Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

12 

- Time 

NA 

1 

2 

3 

4 

5 

6 

7 


DECISION MAKER 









13 

Identified Enemy of Center of Gravity and Attacks 
It. 

NA 

1 

2 

3 

4 

5 

6 

7 

14 

Adjusted to Enemy Situation 

NA 

1 

2 

3 

4 

5 

6 

7 


LEADER / COMMANDER 









15 

Took Charge 

NA 

1 

2 

3 

4 

5 

6 

7 

16 

Maintained Oversight of All Subordinate Units 

NA 

1 

2 

3 

4 

5 

6 

7 


COMMUNICATOR 









17 

Communicated His Plan/Intent to Subordinates 

NA 

1 

2 

3 

4 

5 

6 

7 

18 

Ensured that Subordinates did what they had been 
told 

NA 

1 

2 

3 

4 

5 

6 

7 


TACTICALLY CUNNING 









19 

Used deception and creativity to outwit the enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

20 

Deviated from doctrine when necessary 

NA 

1 

2 

3 

4 

5 

6 

7 





















Source: TBS’s Five Pillars of Leadership (Evaluated by instruetor and survey 
results) 
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APPENDIX O - PRESENTATION OF WORK 


On numerous occasions, we had the opportunity to present this work to the NFS 
Human Factors Focus group and during the Modeling Virtual Environments and 
Simulation (MOVES) Open House. Throughout 2006 and 2007 as we would come up 
with a new idea or completed a component of this thesis, we would ask for some time in 
the schedule to present our findings to the Human Eactors Eocus Group at NFS MOVES 
Institute. This group was a collection of students and professors who were interested in 
the human experience within virtual environments. We received many suggestions and 
constructive comments on our work. Once we had completed Experiment 2, we had the 
privilege of presenting our overall findings at the 2007 MOVES Open House in July 
2007. We were pleased with the response from the audience and were happy to see that 
our work had been of interest to some within the military and from the private sector as 
well. 
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